Nov 01 19:58:01.863578 ip-10-0-136-26 systemd[1]: Starting Kubernetes Kubelet... Nov 01 19:58:02.290120 ip-10-0-136-26 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 19:58:02.290120 ip-10-0-136-26 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 01 19:58:02.290120 ip-10-0-136-26 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 19:58:02.290120 ip-10-0-136-26 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 19:58:02.296708 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296629 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 19:58:02.296708 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296704 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 19:58:02.296708 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296710 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 19:58:02.296799 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296713 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 19:58:02.296799 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296716 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 19:58:02.296799 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296719 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 19:58:02.296799 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296722 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 19:58:02.296799 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296725 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 19:58:02.296799 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296728 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 19:58:02.296799 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296730 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 19:58:02.296799 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296733 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 19:58:02.296799 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296736 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 19:58:02.296799 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296738 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 19:58:02.296799 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296741 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 19:58:02.296799 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296744 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 19:58:02.296799 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296746 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 19:58:02.296799 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296749 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 19:58:02.296799 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296752 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 19:58:02.296799 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296754 2576 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 19:58:02.296799 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296757 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 19:58:02.296799 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296760 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 19:58:02.296799 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296763 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 19:58:02.296799 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296765 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 19:58:02.297264 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296768 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 19:58:02.297264 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296770 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 19:58:02.297264 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296773 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 19:58:02.297264 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296775 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 19:58:02.297264 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296778 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 19:58:02.297264 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296783 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 19:58:02.297264 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296787 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 19:58:02.297264 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296790 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 19:58:02.297264 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296793 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 19:58:02.297264 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296795 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 19:58:02.297264 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296798 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 19:58:02.297264 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296801 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 19:58:02.297264 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296803 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 19:58:02.297264 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296807 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 19:58:02.297264 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296810 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 19:58:02.297264 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296812 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 19:58:02.297264 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296816 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 19:58:02.297264 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296818 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 19:58:02.297264 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296820 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 19:58:02.297730 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296823 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 19:58:02.297730 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296825 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 19:58:02.297730 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296828 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 19:58:02.297730 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296831 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 19:58:02.297730 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296833 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 19:58:02.297730 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296836 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 19:58:02.297730 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296838 2576 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 19:58:02.297730 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296841 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 19:58:02.297730 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296843 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 19:58:02.297730 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296845 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 19:58:02.297730 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296848 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 19:58:02.297730 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296851 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 19:58:02.297730 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296853 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 19:58:02.297730 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296856 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 19:58:02.297730 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296859 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 19:58:02.297730 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296862 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 19:58:02.297730 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296865 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 19:58:02.297730 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296867 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 19:58:02.297730 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296870 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 19:58:02.297730 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296872 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 19:58:02.298210 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296874 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 19:58:02.298210 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296877 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 19:58:02.298210 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296880 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 19:58:02.298210 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296884 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 19:58:02.298210 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296887 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 19:58:02.298210 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296890 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 19:58:02.298210 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296893 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 19:58:02.298210 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296896 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 19:58:02.298210 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296899 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 19:58:02.298210 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296902 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 19:58:02.298210 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296905 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 19:58:02.298210 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296908 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 19:58:02.298210 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296910 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 19:58:02.298210 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296913 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 19:58:02.298210 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296915 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 19:58:02.298210 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296918 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 19:58:02.298210 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296920 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 19:58:02.298210 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296923 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 19:58:02.298210 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296926 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 19:58:02.298675 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296928 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 19:58:02.298675 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296931 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 19:58:02.298675 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296933 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 19:58:02.298675 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296936 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 19:58:02.298675 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296938 2576 feature_gate.go:386] unrecognized feature gate: Example Nov 01 19:58:02.298675 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296941 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 19:58:02.298675 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296943 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 19:58:02.298675 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296947 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 19:58:02.298675 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296950 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 19:58:02.298675 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296952 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 19:58:02.298675 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296955 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 19:58:02.298675 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296958 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 19:58:02.298675 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296960 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 19:58:02.298675 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296963 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 19:58:02.298675 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296966 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 19:58:02.298675 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296968 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 19:58:02.298675 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296971 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 19:58:02.298675 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296974 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 19:58:02.298675 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296977 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 19:58:02.298675 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296979 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 19:58:02.299144 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296982 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 19:58:02.299144 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296984 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 19:58:02.299144 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296987 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 19:58:02.299144 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296990 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 19:58:02.299144 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.296993 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 19:58:02.299144 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297773 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 19:58:02.299144 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297780 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 19:58:02.299144 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297782 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 19:58:02.299144 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297786 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 19:58:02.299144 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297789 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 19:58:02.299144 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297792 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 19:58:02.299144 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297794 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 19:58:02.299144 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297797 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 19:58:02.299144 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297799 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 19:58:02.299144 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297802 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 19:58:02.299144 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297805 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 19:58:02.299144 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297807 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 19:58:02.299144 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297810 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 19:58:02.299144 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297812 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 19:58:02.299603 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297818 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 19:58:02.299603 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297821 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 19:58:02.299603 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297823 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 19:58:02.299603 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297826 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 19:58:02.299603 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297829 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 19:58:02.299603 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297831 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 19:58:02.299603 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297834 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 19:58:02.299603 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297837 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 19:58:02.299603 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297839 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 19:58:02.299603 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297842 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 19:58:02.299603 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297845 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 19:58:02.299603 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297847 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 19:58:02.299603 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297850 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 19:58:02.299603 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297852 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 19:58:02.299603 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297855 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 19:58:02.299603 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297858 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 19:58:02.299603 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297861 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 19:58:02.299603 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297863 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 19:58:02.299603 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297866 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 19:58:02.300050 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297869 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 19:58:02.300050 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297872 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 19:58:02.300050 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297874 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 19:58:02.300050 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297877 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 19:58:02.300050 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297879 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 19:58:02.300050 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297882 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 19:58:02.300050 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297884 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 19:58:02.300050 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297887 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 19:58:02.300050 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297889 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 19:58:02.300050 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297892 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 19:58:02.300050 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297894 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 19:58:02.300050 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297897 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 19:58:02.300050 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297900 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 19:58:02.300050 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297906 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 19:58:02.300050 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297910 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 19:58:02.300050 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297913 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 19:58:02.300050 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297916 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 19:58:02.300050 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297919 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 19:58:02.300050 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297938 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 19:58:02.300506 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297942 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 19:58:02.300506 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297945 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 19:58:02.300506 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297947 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 19:58:02.300506 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297951 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 19:58:02.300506 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297953 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 19:58:02.300506 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297956 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 19:58:02.300506 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297959 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 19:58:02.300506 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297961 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 19:58:02.300506 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297964 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 19:58:02.300506 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297966 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 19:58:02.300506 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297969 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 19:58:02.300506 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297971 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 19:58:02.300506 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297974 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 19:58:02.300506 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297976 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 19:58:02.300506 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297979 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 19:58:02.300506 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297982 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 19:58:02.300506 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297984 2576 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 19:58:02.300506 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297987 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 19:58:02.300506 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297990 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 19:58:02.300506 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297992 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 19:58:02.300986 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297994 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 19:58:02.300986 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297997 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 19:58:02.300986 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.297999 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 19:58:02.300986 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298002 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 19:58:02.300986 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298004 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 19:58:02.300986 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298007 2576 feature_gate.go:386] unrecognized feature gate: Example Nov 01 19:58:02.300986 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298011 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 19:58:02.300986 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298014 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 19:58:02.300986 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298017 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 19:58:02.300986 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298019 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 19:58:02.300986 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298022 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 19:58:02.300986 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298024 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 19:58:02.300986 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298027 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 19:58:02.300986 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298029 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 19:58:02.300986 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298032 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 19:58:02.300986 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298034 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 19:58:02.300986 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298037 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 19:58:02.300986 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298041 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 19:58:02.300986 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298045 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 19:58:02.300986 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298048 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 19:58:02.301461 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298050 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 19:58:02.301461 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298053 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 19:58:02.301461 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298064 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 19:58:02.301461 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298067 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 19:58:02.301461 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298070 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 19:58:02.301461 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298072 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 19:58:02.301461 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298076 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 19:58:02.301461 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298079 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 19:58:02.301461 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298081 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 19:58:02.301461 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298084 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 19:58:02.301461 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298088 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 19:58:02.301461 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298090 2576 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 19:58:02.301461 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298093 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 19:58:02.301461 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298095 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 19:58:02.301461 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298638 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 19:58:02.301461 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298645 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 19:58:02.301461 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298648 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 19:58:02.301461 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298651 2576 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 19:58:02.301461 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298659 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 19:58:02.301903 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298663 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 19:58:02.301903 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298666 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 19:58:02.301903 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298669 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 19:58:02.301903 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298672 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 19:58:02.301903 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298674 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 19:58:02.301903 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298677 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 19:58:02.301903 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298679 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 19:58:02.301903 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298682 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 19:58:02.301903 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298685 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 19:58:02.301903 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298688 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 19:58:02.301903 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298690 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 19:58:02.301903 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298693 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 19:58:02.301903 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298696 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 19:58:02.301903 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298699 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 19:58:02.301903 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298701 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 19:58:02.301903 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298704 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 19:58:02.301903 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298706 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 19:58:02.301903 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298709 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 19:58:02.301903 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298712 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 19:58:02.301903 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298715 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 19:58:02.302380 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298717 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 19:58:02.302380 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298720 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 19:58:02.302380 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298722 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 19:58:02.302380 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298725 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 19:58:02.302380 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298727 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 19:58:02.302380 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298730 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 19:58:02.302380 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298732 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 19:58:02.302380 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298735 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 19:58:02.302380 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298737 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 19:58:02.302380 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298740 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 19:58:02.302380 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298742 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 19:58:02.302380 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298746 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 19:58:02.302380 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298749 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 19:58:02.302380 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298751 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 19:58:02.302380 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298754 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 19:58:02.302380 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298757 2576 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 19:58:02.302380 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298759 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 19:58:02.302380 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298762 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 19:58:02.302380 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298764 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 19:58:02.302380 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298767 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 19:58:02.302844 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298769 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 19:58:02.302844 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298772 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 19:58:02.302844 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298774 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 19:58:02.302844 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298777 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 19:58:02.302844 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298779 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 19:58:02.302844 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298782 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 19:58:02.302844 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298785 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 19:58:02.302844 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298787 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 19:58:02.302844 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298789 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 19:58:02.302844 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298792 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 19:58:02.302844 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298794 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 19:58:02.302844 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298797 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 19:58:02.302844 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298800 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 19:58:02.302844 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298802 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 19:58:02.302844 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298805 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 19:58:02.302844 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298808 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 19:58:02.302844 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298810 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 19:58:02.302844 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298813 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 19:58:02.302844 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298815 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 19:58:02.302844 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298818 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 19:58:02.303349 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298820 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 19:58:02.303349 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298823 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 19:58:02.303349 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298825 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 19:58:02.303349 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298830 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 19:58:02.303349 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298833 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 19:58:02.303349 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298836 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 19:58:02.303349 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298838 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 19:58:02.303349 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298842 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 19:58:02.303349 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298846 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 19:58:02.303349 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298849 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 19:58:02.303349 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298852 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 19:58:02.303349 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298854 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 19:58:02.303349 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298857 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 19:58:02.303349 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298859 2576 feature_gate.go:386] unrecognized feature gate: Example Nov 01 19:58:02.303349 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298862 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 19:58:02.303349 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298864 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 19:58:02.303349 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298867 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 19:58:02.303349 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298870 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 19:58:02.303349 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298872 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 19:58:02.303872 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298875 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 19:58:02.303872 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298877 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 19:58:02.303872 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298880 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 19:58:02.303872 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298882 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 19:58:02.303872 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298885 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 19:58:02.303872 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298887 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 19:58:02.303872 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298890 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 19:58:02.303872 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298893 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 19:58:02.303872 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298895 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 19:58:02.303872 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298898 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 19:58:02.303872 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298900 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 19:58:02.303872 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298903 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 19:58:02.303872 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298905 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 19:58:02.303872 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298908 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 19:58:02.303872 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298910 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 19:58:02.303872 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298917 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 19:58:02.303872 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298921 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 19:58:02.303872 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298924 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 19:58:02.303872 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298927 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 19:58:02.304367 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298930 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 19:58:02.304367 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298932 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 19:58:02.304367 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.298935 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 19:58:02.304367 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300062 2576 flags.go:64] FLAG: --address="0.0.0.0" Nov 01 19:58:02.304367 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300071 2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 01 19:58:02.304367 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300078 2576 flags.go:64] FLAG: --anonymous-auth="true" Nov 01 19:58:02.304367 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300083 2576 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 01 19:58:02.304367 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300087 2576 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 01 19:58:02.304367 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300091 2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 01 19:58:02.304367 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300095 2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 01 19:58:02.304367 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300099 2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 01 19:58:02.304367 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300103 2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 01 19:58:02.304367 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300106 2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 01 19:58:02.304367 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300110 2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 01 19:58:02.304367 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300114 2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 01 19:58:02.304367 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300117 2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 01 19:58:02.304367 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300120 2576 flags.go:64] FLAG: --cgroup-root="" Nov 01 19:58:02.304367 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300123 2576 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 01 19:58:02.304367 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300126 2576 flags.go:64] FLAG: --client-ca-file="" Nov 01 19:58:02.304367 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300130 2576 flags.go:64] FLAG: --cloud-provider="external" Nov 01 19:58:02.304367 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300133 2576 flags.go:64] FLAG: --cluster-dns="[]" Nov 01 19:58:02.304367 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300137 2576 flags.go:64] FLAG: --cluster-domain="" Nov 01 19:58:02.304367 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300140 2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 01 19:58:02.304950 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300143 2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 01 19:58:02.304950 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300147 2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 01 19:58:02.304950 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300151 2576 flags.go:64] FLAG: --container-log-max-files="5" Nov 01 19:58:02.304950 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300154 2576 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 01 19:58:02.304950 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300158 2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 01 19:58:02.304950 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300162 2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 01 19:58:02.304950 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300166 2576 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 01 19:58:02.304950 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300169 2576 flags.go:64] FLAG: --contention-profiling="false" Nov 01 19:58:02.304950 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300172 2576 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 01 19:58:02.304950 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300175 2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 01 19:58:02.304950 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300178 2576 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 01 19:58:02.304950 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300181 2576 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 01 19:58:02.304950 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300185 2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 01 19:58:02.304950 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300188 2576 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 01 19:58:02.304950 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300191 2576 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 01 19:58:02.304950 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300194 2576 flags.go:64] FLAG: --enable-load-reader="false" Nov 01 19:58:02.304950 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300197 2576 flags.go:64] FLAG: --enable-server="true" Nov 01 19:58:02.304950 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300200 2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 01 19:58:02.304950 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300204 2576 flags.go:64] FLAG: --event-burst="100" Nov 01 19:58:02.304950 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300207 2576 flags.go:64] FLAG: --event-qps="50" Nov 01 19:58:02.304950 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300210 2576 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 01 19:58:02.304950 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300213 2576 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 01 19:58:02.304950 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300216 2576 flags.go:64] FLAG: --eviction-hard="" Nov 01 19:58:02.304950 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300220 2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 01 19:58:02.305554 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300223 2576 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 01 19:58:02.305554 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300226 2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 01 19:58:02.305554 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300229 2576 flags.go:64] FLAG: --eviction-soft="" Nov 01 19:58:02.305554 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300232 2576 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 01 19:58:02.305554 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300235 2576 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 01 19:58:02.305554 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300238 2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 01 19:58:02.305554 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300241 2576 flags.go:64] FLAG: --experimental-mounter-path="" Nov 01 19:58:02.305554 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300244 2576 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 01 19:58:02.305554 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300247 2576 flags.go:64] FLAG: --fail-swap-on="true" Nov 01 19:58:02.305554 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300250 2576 flags.go:64] FLAG: --feature-gates="" Nov 01 19:58:02.305554 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300254 2576 flags.go:64] FLAG: --file-check-frequency="20s" Nov 01 19:58:02.305554 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300257 2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 01 19:58:02.305554 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300260 2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 01 19:58:02.305554 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300264 2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 01 19:58:02.305554 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300267 2576 flags.go:64] FLAG: --healthz-port="10248" Nov 01 19:58:02.305554 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300271 2576 flags.go:64] FLAG: --help="false" Nov 01 19:58:02.305554 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300274 2576 flags.go:64] FLAG: --hostname-override="ip-10-0-136-26.us-east-2.compute.internal" Nov 01 19:58:02.305554 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300277 2576 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 01 19:58:02.305554 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300280 2576 flags.go:64] FLAG: --http-check-frequency="20s" Nov 01 19:58:02.305554 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300294 2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 01 19:58:02.305554 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300298 2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 01 19:58:02.305554 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300302 2576 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 01 19:58:02.305554 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300305 2576 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 01 19:58:02.306096 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300308 2576 flags.go:64] FLAG: --image-service-endpoint="" Nov 01 19:58:02.306096 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300311 2576 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 01 19:58:02.306096 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300314 2576 flags.go:64] FLAG: --kube-api-burst="100" Nov 01 19:58:02.306096 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300317 2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 01 19:58:02.306096 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300320 2576 flags.go:64] FLAG: --kube-api-qps="50" Nov 01 19:58:02.306096 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300323 2576 flags.go:64] FLAG: --kube-reserved="" Nov 01 19:58:02.306096 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300326 2576 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 01 19:58:02.306096 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300329 2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 01 19:58:02.306096 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300332 2576 flags.go:64] FLAG: --kubelet-cgroups="" Nov 01 19:58:02.306096 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300335 2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 01 19:58:02.306096 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300338 2576 flags.go:64] FLAG: --lock-file="" Nov 01 19:58:02.306096 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300341 2576 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 01 19:58:02.306096 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300344 2576 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 01 19:58:02.306096 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300347 2576 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 01 19:58:02.306096 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300352 2576 flags.go:64] FLAG: --log-json-split-stream="false" Nov 01 19:58:02.306096 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300355 2576 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 01 19:58:02.306096 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300359 2576 flags.go:64] FLAG: --log-text-split-stream="false" Nov 01 19:58:02.306096 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300362 2576 flags.go:64] FLAG: --logging-format="text" Nov 01 19:58:02.306096 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300365 2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 01 19:58:02.306096 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300368 2576 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 01 19:58:02.306096 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300371 2576 flags.go:64] FLAG: --manifest-url="" Nov 01 19:58:02.306096 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300374 2576 flags.go:64] FLAG: --manifest-url-header="" Nov 01 19:58:02.306096 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300379 2576 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 01 19:58:02.306096 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300383 2576 flags.go:64] FLAG: --max-open-files="1000000" Nov 01 19:58:02.306096 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300387 2576 flags.go:64] FLAG: --max-pods="110" Nov 01 19:58:02.306695 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300390 2576 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 01 19:58:02.306695 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300393 2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 01 19:58:02.306695 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300396 2576 flags.go:64] FLAG: --memory-manager-policy="None" Nov 01 19:58:02.306695 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300398 2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 01 19:58:02.306695 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300402 2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 01 19:58:02.306695 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300404 2576 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 01 19:58:02.306695 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300407 2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 01 19:58:02.306695 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300414 2576 flags.go:64] FLAG: --node-status-max-images="50" Nov 01 19:58:02.306695 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300417 2576 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 01 19:58:02.306695 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300420 2576 flags.go:64] FLAG: --oom-score-adj="-999" Nov 01 19:58:02.306695 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300423 2576 flags.go:64] FLAG: --pod-cidr="" Nov 01 19:58:02.306695 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300426 2576 flags.go:64] FLAG: --pod-manifest-path="" Nov 01 19:58:02.306695 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300429 2576 flags.go:64] FLAG: --pod-max-pids="-1" Nov 01 19:58:02.306695 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300432 2576 flags.go:64] FLAG: --pods-per-core="0" Nov 01 19:58:02.306695 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300435 2576 flags.go:64] FLAG: --port="10250" Nov 01 19:58:02.306695 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300438 2576 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 01 19:58:02.306695 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300441 2576 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-04d4951a5c963ee10" Nov 01 19:58:02.306695 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300444 2576 flags.go:64] FLAG: --qos-reserved="" Nov 01 19:58:02.306695 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300447 2576 flags.go:64] FLAG: --read-only-port="10255" Nov 01 19:58:02.306695 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300450 2576 flags.go:64] FLAG: --register-node="true" Nov 01 19:58:02.306695 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300453 2576 flags.go:64] FLAG: --register-with-taints="" Nov 01 19:58:02.306695 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300457 2576 flags.go:64] FLAG: --registry-burst="10" Nov 01 19:58:02.306695 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300460 2576 flags.go:64] FLAG: --registry-qps="5" Nov 01 19:58:02.306695 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300462 2576 flags.go:64] FLAG: --reserved-cpus="" Nov 01 19:58:02.306695 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300466 2576 flags.go:64] FLAG: --reserved-memory="" Nov 01 19:58:02.306695 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300470 2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 01 19:58:02.307343 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300473 2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 01 19:58:02.307343 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300476 2576 flags.go:64] FLAG: --rotate-certificates="false" Nov 01 19:58:02.307343 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300478 2576 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 01 19:58:02.307343 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300482 2576 flags.go:64] FLAG: --runonce="false" Nov 01 19:58:02.307343 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300486 2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 01 19:58:02.307343 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300489 2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 01 19:58:02.307343 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300492 2576 flags.go:64] FLAG: --seccomp-default="false" Nov 01 19:58:02.307343 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300495 2576 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 01 19:58:02.307343 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300498 2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 01 19:58:02.307343 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300501 2576 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 01 19:58:02.307343 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300504 2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 01 19:58:02.307343 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300507 2576 flags.go:64] FLAG: --storage-driver-password="root" Nov 01 19:58:02.307343 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300510 2576 flags.go:64] FLAG: --storage-driver-secure="false" Nov 01 19:58:02.307343 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300513 2576 flags.go:64] FLAG: --storage-driver-table="stats" Nov 01 19:58:02.307343 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300516 2576 flags.go:64] FLAG: --storage-driver-user="root" Nov 01 19:58:02.307343 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300519 2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 01 19:58:02.307343 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300522 2576 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 01 19:58:02.307343 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300525 2576 flags.go:64] FLAG: --system-cgroups="" Nov 01 19:58:02.307343 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300528 2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 01 19:58:02.307343 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300533 2576 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 01 19:58:02.307343 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300537 2576 flags.go:64] FLAG: --tls-cert-file="" Nov 01 19:58:02.307343 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300539 2576 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 01 19:58:02.307343 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300544 2576 flags.go:64] FLAG: --tls-min-version="" Nov 01 19:58:02.307343 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300546 2576 flags.go:64] FLAG: --tls-private-key-file="" Nov 01 19:58:02.307343 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300549 2576 flags.go:64] FLAG: --topology-manager-policy="none" Nov 01 19:58:02.307939 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300552 2576 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 01 19:58:02.307939 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300555 2576 flags.go:64] FLAG: --topology-manager-scope="container" Nov 01 19:58:02.307939 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300558 2576 flags.go:64] FLAG: --v="2" Nov 01 19:58:02.307939 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300562 2576 flags.go:64] FLAG: --version="false" Nov 01 19:58:02.307939 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300566 2576 flags.go:64] FLAG: --vmodule="" Nov 01 19:58:02.307939 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300570 2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 01 19:58:02.307939 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.300574 2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 01 19:58:02.307939 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300701 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 19:58:02.307939 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300705 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 19:58:02.307939 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300707 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 19:58:02.307939 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300712 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 19:58:02.307939 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300716 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 19:58:02.307939 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300718 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 19:58:02.307939 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300721 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 19:58:02.307939 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300723 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 19:58:02.307939 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300726 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 19:58:02.307939 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300728 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 19:58:02.307939 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300731 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 19:58:02.307939 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300734 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 19:58:02.307939 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300736 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 19:58:02.307939 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300740 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 19:58:02.308536 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300743 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 19:58:02.308536 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300746 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 19:58:02.308536 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300748 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 19:58:02.308536 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300751 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 19:58:02.308536 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300753 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 19:58:02.308536 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300756 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 19:58:02.308536 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300759 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 19:58:02.308536 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300761 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 19:58:02.308536 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300764 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 19:58:02.308536 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300766 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 19:58:02.308536 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300769 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 19:58:02.308536 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300772 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 19:58:02.308536 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300775 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 19:58:02.308536 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300777 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 19:58:02.308536 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300780 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 19:58:02.308536 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300782 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 19:58:02.308536 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300785 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 19:58:02.308536 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300788 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 19:58:02.308536 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300791 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 19:58:02.308536 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300793 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 19:58:02.309012 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300796 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 19:58:02.309012 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300800 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 19:58:02.309012 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300804 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 19:58:02.309012 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300807 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 19:58:02.309012 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300810 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 19:58:02.309012 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300812 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 19:58:02.309012 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300816 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 19:58:02.309012 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300818 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 19:58:02.309012 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300821 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 19:58:02.309012 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300823 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 19:58:02.309012 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300826 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 19:58:02.309012 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300828 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 19:58:02.309012 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300831 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 19:58:02.309012 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300834 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 19:58:02.309012 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300836 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 19:58:02.309012 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300839 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 19:58:02.309012 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300841 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 19:58:02.309012 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300844 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 19:58:02.309012 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300846 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 19:58:02.309012 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300849 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 19:58:02.309518 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300852 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 19:58:02.309518 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300855 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 19:58:02.309518 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300859 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 19:58:02.309518 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300862 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 19:58:02.309518 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300864 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 19:58:02.309518 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300867 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 19:58:02.309518 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300870 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 19:58:02.309518 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300872 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 19:58:02.309518 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300875 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 19:58:02.309518 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300877 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 19:58:02.309518 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300880 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 19:58:02.309518 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300883 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 19:58:02.309518 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300886 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 19:58:02.309518 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300902 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 19:58:02.309518 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300906 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 19:58:02.309518 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300908 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 19:58:02.309518 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300911 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 19:58:02.309518 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300914 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 19:58:02.309518 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300916 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 19:58:02.309981 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300919 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 19:58:02.309981 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300922 2576 feature_gate.go:386] unrecognized feature gate: Example Nov 01 19:58:02.309981 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300925 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 19:58:02.309981 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300927 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 19:58:02.309981 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300930 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 19:58:02.309981 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300934 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 19:58:02.309981 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300938 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 19:58:02.309981 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300941 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 19:58:02.309981 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300944 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 19:58:02.309981 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300947 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 19:58:02.309981 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300950 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 19:58:02.309981 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300952 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 19:58:02.309981 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300955 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 19:58:02.309981 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300957 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 19:58:02.309981 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300960 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 19:58:02.309981 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300962 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 19:58:02.309981 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300965 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 19:58:02.309981 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300968 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 19:58:02.309981 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300970 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 19:58:02.310446 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300973 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 19:58:02.310446 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300975 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 19:58:02.310446 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300978 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 19:58:02.310446 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300980 2576 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 19:58:02.310446 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300983 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 19:58:02.310446 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300985 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 19:58:02.310446 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300995 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 19:58:02.310446 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.300998 2576 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 19:58:02.310446 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301003 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 19:58:02.310446 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301006 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 19:58:02.310446 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301009 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 19:58:02.310446 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301011 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 19:58:02.310446 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301014 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 19:58:02.310446 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301017 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 19:58:02.310446 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.301043 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 19:58:02.310446 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301091 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 19:58:02.310862 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301096 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 19:58:02.310862 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301099 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 19:58:02.310862 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301102 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 19:58:02.310862 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301105 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 19:58:02.310862 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301108 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 19:58:02.310862 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301110 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 19:58:02.310862 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301113 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 19:58:02.310862 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301116 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 19:58:02.310862 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301119 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 19:58:02.310862 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301121 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 19:58:02.310862 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301124 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 19:58:02.310862 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301127 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 19:58:02.310862 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301130 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 19:58:02.310862 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301133 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 19:58:02.310862 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301138 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 19:58:02.310862 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301141 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 19:58:02.310862 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301144 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 19:58:02.310862 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301147 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 19:58:02.310862 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301149 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 19:58:02.311323 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301152 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 19:58:02.311323 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301154 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 19:58:02.311323 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301157 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 19:58:02.311323 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301160 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 19:58:02.311323 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301163 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 19:58:02.311323 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301166 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 19:58:02.311323 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301170 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 19:58:02.311323 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301173 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 19:58:02.311323 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301175 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 19:58:02.311323 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301178 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 19:58:02.311323 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301180 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 19:58:02.311323 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301184 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 19:58:02.311323 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301186 2576 feature_gate.go:386] unrecognized feature gate: Example Nov 01 19:58:02.311323 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301189 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 19:58:02.311323 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301191 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 19:58:02.311323 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301194 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 19:58:02.311323 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301197 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 19:58:02.311323 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301201 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 19:58:02.311323 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301205 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 19:58:02.311766 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301208 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 19:58:02.311766 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301210 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 19:58:02.311766 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301213 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 19:58:02.311766 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301216 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 19:58:02.311766 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301218 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 19:58:02.311766 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301221 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 19:58:02.311766 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301223 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 19:58:02.311766 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301226 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 19:58:02.311766 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301228 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 19:58:02.311766 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301231 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 19:58:02.311766 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301233 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 19:58:02.311766 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301236 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 19:58:02.311766 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301238 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 19:58:02.311766 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301241 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 19:58:02.311766 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301244 2576 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 19:58:02.311766 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301247 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 19:58:02.311766 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301249 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 19:58:02.311766 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301252 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 19:58:02.311766 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301255 2576 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 19:58:02.312205 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301260 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 19:58:02.312205 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301262 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 19:58:02.312205 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301265 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 19:58:02.312205 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301267 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 19:58:02.312205 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301271 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 19:58:02.312205 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301273 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 19:58:02.312205 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301276 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 19:58:02.312205 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301278 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 19:58:02.312205 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301281 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 19:58:02.312205 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301294 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 19:58:02.312205 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301297 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 19:58:02.312205 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301299 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 19:58:02.312205 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301302 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 19:58:02.312205 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301305 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 19:58:02.312205 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301307 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 19:58:02.312205 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301324 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 19:58:02.312205 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301327 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 19:58:02.312205 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301331 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 19:58:02.312205 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301334 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 19:58:02.312205 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301336 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 19:58:02.312674 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301339 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 19:58:02.312674 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301342 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 19:58:02.312674 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301344 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 19:58:02.312674 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301347 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 19:58:02.312674 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301349 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 19:58:02.312674 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301352 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 19:58:02.312674 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301355 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 19:58:02.312674 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301357 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 19:58:02.312674 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301360 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 19:58:02.312674 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301363 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 19:58:02.312674 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301365 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 19:58:02.312674 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301368 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 19:58:02.312674 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301374 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 19:58:02.312674 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301376 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 19:58:02.312674 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301379 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 19:58:02.312674 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301382 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 19:58:02.312674 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301384 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 19:58:02.312674 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301387 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 19:58:02.312674 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301390 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 19:58:02.312674 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301392 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 19:58:02.313136 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301395 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 19:58:02.313136 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301397 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 19:58:02.313136 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301400 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 19:58:02.313136 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301402 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 19:58:02.313136 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301406 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 19:58:02.313136 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301408 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 19:58:02.313136 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301411 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 19:58:02.313136 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.301413 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 19:58:02.313136 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.308355 2576 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 01 19:58:02.313136 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.308371 2576 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 01 19:58:02.313136 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.308387 2576 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 01 19:58:02.313136 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.308392 2576 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 01 19:58:02.313136 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308466 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 19:58:02.313136 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308472 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 19:58:02.313136 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308476 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 19:58:02.313136 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308479 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 19:58:02.313136 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308482 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 19:58:02.313136 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308484 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 19:58:02.313136 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308487 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 19:58:02.313136 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308489 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 19:58:02.313609 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308492 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 19:58:02.313609 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308495 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 19:58:02.313609 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308498 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 19:58:02.313609 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308502 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 19:58:02.313609 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308507 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 19:58:02.313609 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308510 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 19:58:02.313609 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308513 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 19:58:02.313609 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308516 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 19:58:02.313609 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308520 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 19:58:02.313609 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308522 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 19:58:02.313609 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308525 2576 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 19:58:02.313609 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308528 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 19:58:02.313609 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308531 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 19:58:02.313609 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308533 2576 feature_gate.go:386] unrecognized feature gate: Example Nov 01 19:58:02.313609 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308536 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 19:58:02.313609 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308539 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 19:58:02.313609 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308542 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 19:58:02.313609 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308544 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 19:58:02.313609 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308547 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 19:58:02.314065 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308550 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 19:58:02.314065 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308552 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 19:58:02.314065 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308556 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 19:58:02.314065 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308559 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 19:58:02.314065 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308562 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 19:58:02.314065 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308565 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 19:58:02.314065 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308568 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 19:58:02.314065 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308570 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 19:58:02.314065 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308573 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 19:58:02.314065 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308576 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 19:58:02.314065 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308578 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 19:58:02.314065 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308581 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 19:58:02.314065 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308585 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 19:58:02.314065 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308589 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 19:58:02.314065 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308592 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 19:58:02.314065 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308595 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 19:58:02.314065 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308598 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 19:58:02.314065 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308601 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 19:58:02.314065 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308604 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 19:58:02.314558 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308607 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 19:58:02.314558 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308609 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 19:58:02.314558 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308612 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 19:58:02.314558 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308614 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 19:58:02.314558 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308617 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 19:58:02.314558 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308620 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 19:58:02.314558 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308623 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 19:58:02.314558 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308625 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 19:58:02.314558 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308628 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 19:58:02.314558 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308631 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 19:58:02.314558 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308633 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 19:58:02.314558 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308635 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 19:58:02.314558 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308638 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 19:58:02.314558 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308641 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 19:58:02.314558 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308643 2576 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 19:58:02.314558 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308646 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 19:58:02.314558 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308649 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 19:58:02.314558 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308652 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 19:58:02.314558 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308655 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 19:58:02.314558 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308657 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 19:58:02.315030 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308660 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 19:58:02.315030 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308662 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 19:58:02.315030 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308665 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 19:58:02.315030 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308668 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 19:58:02.315030 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308670 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 19:58:02.315030 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308673 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 19:58:02.315030 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308675 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 19:58:02.315030 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308678 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 19:58:02.315030 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308680 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 19:58:02.315030 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308683 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 19:58:02.315030 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308685 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 19:58:02.315030 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308688 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 19:58:02.315030 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308691 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 19:58:02.315030 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308694 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 19:58:02.315030 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308696 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 19:58:02.315030 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308699 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 19:58:02.315030 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308701 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 19:58:02.315030 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308704 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 19:58:02.315030 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308706 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 19:58:02.315030 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308709 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 19:58:02.315547 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308711 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 19:58:02.315547 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308714 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 19:58:02.315547 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308716 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 19:58:02.315547 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308719 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 19:58:02.315547 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308721 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 19:58:02.315547 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308724 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 19:58:02.315547 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308726 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 19:58:02.315547 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308730 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 19:58:02.315547 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308732 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 19:58:02.315547 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308735 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 19:58:02.315547 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308738 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 19:58:02.315547 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308741 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 19:58:02.315547 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308743 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 19:58:02.315547 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308746 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 19:58:02.315547 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308748 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 19:58:02.315547 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308751 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 19:58:02.315547 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308753 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 19:58:02.315547 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308756 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 19:58:02.315547 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308758 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 19:58:02.315978 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308761 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 19:58:02.315978 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.308787 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 19:58:02.315978 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308920 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 19:58:02.315978 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308924 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 19:58:02.315978 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308927 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 19:58:02.315978 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308930 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 19:58:02.315978 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308933 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 19:58:02.315978 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308936 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 19:58:02.315978 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308938 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 19:58:02.315978 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308941 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 19:58:02.315978 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308943 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 19:58:02.315978 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308946 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 19:58:02.315978 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308948 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 19:58:02.315978 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308950 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 19:58:02.315978 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308953 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 19:58:02.315978 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308955 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 19:58:02.316364 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308958 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 19:58:02.316364 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308960 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 19:58:02.316364 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308963 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 19:58:02.316364 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308965 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 19:58:02.316364 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308968 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 19:58:02.316364 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308971 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 19:58:02.316364 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308973 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 19:58:02.316364 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308976 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 19:58:02.316364 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308978 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 19:58:02.316364 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308981 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 19:58:02.316364 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308984 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 19:58:02.316364 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308986 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 19:58:02.316364 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308989 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 19:58:02.316364 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308991 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 19:58:02.316364 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308994 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 19:58:02.316364 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308996 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 19:58:02.316364 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.308999 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 19:58:02.316364 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309001 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 19:58:02.316364 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309005 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 19:58:02.317549 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309008 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 19:58:02.317549 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309011 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 19:58:02.317549 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309014 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 19:58:02.317549 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309017 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 19:58:02.317549 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309020 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 19:58:02.317549 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309023 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 19:58:02.317549 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309025 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 19:58:02.317549 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309028 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 19:58:02.317549 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309030 2576 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 19:58:02.317549 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309033 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 19:58:02.317549 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309035 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 19:58:02.317549 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309038 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 19:58:02.317549 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309040 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 19:58:02.317549 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309043 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 19:58:02.317549 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309045 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 19:58:02.317549 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309048 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 19:58:02.317549 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309050 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 19:58:02.317549 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309054 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 19:58:02.317549 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309058 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 19:58:02.317990 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309062 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 19:58:02.317990 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309065 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 19:58:02.317990 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309067 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 19:58:02.317990 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309070 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 19:58:02.317990 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309073 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 19:58:02.317990 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309076 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 19:58:02.317990 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309078 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 19:58:02.317990 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309081 2576 feature_gate.go:386] unrecognized feature gate: Example Nov 01 19:58:02.317990 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309083 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 19:58:02.317990 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309086 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 19:58:02.317990 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309088 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 19:58:02.317990 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309091 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 19:58:02.317990 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309093 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 19:58:02.317990 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309096 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 19:58:02.317990 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309098 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 19:58:02.317990 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309101 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 19:58:02.317990 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309103 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 19:58:02.317990 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309105 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 19:58:02.317990 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309108 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 19:58:02.317990 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309111 2576 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 19:58:02.318728 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309113 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 19:58:02.318728 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309116 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 19:58:02.318728 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309118 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 19:58:02.318728 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309121 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 19:58:02.318728 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309123 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 19:58:02.318728 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309126 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 19:58:02.318728 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309128 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 19:58:02.318728 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309131 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 19:58:02.318728 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309133 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 19:58:02.318728 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309136 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 19:58:02.318728 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309138 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 19:58:02.318728 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309141 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 19:58:02.318728 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309144 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 19:58:02.318728 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309146 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 19:58:02.318728 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309149 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 19:58:02.318728 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309151 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 19:58:02.318728 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309154 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 19:58:02.318728 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309156 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 19:58:02.318728 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309159 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 19:58:02.318728 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309162 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 19:58:02.319299 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309164 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 19:58:02.319299 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309167 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 19:58:02.319299 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309169 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 19:58:02.319299 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309172 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 19:58:02.319299 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309174 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 19:58:02.319299 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309177 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 19:58:02.319299 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309179 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 19:58:02.319299 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309181 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 19:58:02.319299 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309184 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 19:58:02.319299 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309186 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 19:58:02.319299 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309189 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 19:58:02.319299 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309192 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 19:58:02.319299 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309194 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 19:58:02.319299 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309197 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 19:58:02.319299 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.309219 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 19:58:02.319656 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309248 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 19:58:02.319656 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309253 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 19:58:02.319656 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309257 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 19:58:02.319656 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309259 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 19:58:02.319656 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309262 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 19:58:02.319656 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309265 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 19:58:02.319656 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309267 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 19:58:02.319656 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309270 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 19:58:02.319656 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309273 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 19:58:02.319656 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309276 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 19:58:02.319656 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309278 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 19:58:02.319656 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309281 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 19:58:02.319656 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309300 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 19:58:02.319656 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309303 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 19:58:02.319656 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309306 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 19:58:02.319656 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309308 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 19:58:02.319656 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309311 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 19:58:02.319656 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309313 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 19:58:02.319656 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309316 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 19:58:02.320212 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309319 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 19:58:02.320212 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309321 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 19:58:02.320212 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309324 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 19:58:02.320212 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309326 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 19:58:02.320212 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309329 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 19:58:02.320212 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309331 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 19:58:02.320212 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309334 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 19:58:02.320212 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309336 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 19:58:02.320212 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309339 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 19:58:02.320212 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309341 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 19:58:02.320212 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309344 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 19:58:02.320212 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309347 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 19:58:02.320212 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309350 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 19:58:02.320212 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309353 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 19:58:02.320212 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309355 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 19:58:02.320212 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309358 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 19:58:02.320212 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309360 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 19:58:02.320212 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309363 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 19:58:02.320212 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309365 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 19:58:02.320697 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309368 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 19:58:02.320697 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309370 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 19:58:02.320697 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309373 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 19:58:02.320697 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309375 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 19:58:02.320697 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309379 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 19:58:02.320697 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309381 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 19:58:02.320697 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309384 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 19:58:02.320697 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309386 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 19:58:02.320697 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309389 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 19:58:02.320697 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309392 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 19:58:02.320697 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309394 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 19:58:02.320697 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309396 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 19:58:02.320697 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309399 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 19:58:02.320697 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309401 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 19:58:02.320697 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309404 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 19:58:02.320697 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309406 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 19:58:02.320697 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309409 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 19:58:02.320697 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309411 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 19:58:02.320697 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309414 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 19:58:02.320697 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309416 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 19:58:02.321294 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309419 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 19:58:02.321294 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309421 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 19:58:02.321294 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309424 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 19:58:02.321294 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309426 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 19:58:02.321294 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309429 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 19:58:02.321294 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309433 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 19:58:02.321294 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309437 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 19:58:02.321294 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309440 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 19:58:02.321294 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309443 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 19:58:02.321294 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309445 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 19:58:02.321294 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309448 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 19:58:02.321294 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309450 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 19:58:02.321294 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309453 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 19:58:02.321294 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309455 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 19:58:02.321294 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309458 2576 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 19:58:02.321294 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309461 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 19:58:02.321294 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309463 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 19:58:02.321294 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309466 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 19:58:02.321294 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309468 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 19:58:02.322026 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309470 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 19:58:02.322026 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309473 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 19:58:02.322026 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309476 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 19:58:02.322026 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309478 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 19:58:02.322026 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309481 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 19:58:02.322026 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309484 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 19:58:02.322026 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309488 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 19:58:02.322026 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309491 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 19:58:02.322026 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309493 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 19:58:02.322026 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309496 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 19:58:02.322026 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309498 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 19:58:02.322026 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309501 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 19:58:02.322026 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309503 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 19:58:02.322026 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309506 2576 feature_gate.go:386] unrecognized feature gate: Example Nov 01 19:58:02.322026 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309508 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 19:58:02.322026 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309511 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 19:58:02.322026 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309513 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 19:58:02.322026 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309516 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 19:58:02.322026 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309519 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 19:58:02.322514 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309522 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 19:58:02.322514 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309525 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 19:58:02.322514 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309527 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 19:58:02.322514 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309530 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 19:58:02.322514 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309532 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 19:58:02.322514 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309535 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 19:58:02.322514 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309540 2576 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 19:58:02.322514 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309545 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 19:58:02.322514 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309547 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 19:58:02.322514 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:02.309550 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 19:58:02.322514 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.309598 2576 server.go:957] "Client rotation is on, will bootstrap in background" Nov 01 19:58:02.322514 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.311870 2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 01 19:58:02.322514 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.312984 2576 server.go:1014] "Starting client certificate rotation" Nov 01 19:58:02.322514 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.313079 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 19:58:02.322514 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.313964 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 19:58:02.322514 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.315248 2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 19:58:02.323209 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.323191 2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 19:58:02.337313 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.337296 2576 log.go:25] "Validated CRI v1 runtime API" Nov 01 19:58:02.341341 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.341326 2576 log.go:25] "Validated CRI v1 image API" Nov 01 19:58:02.342109 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.342082 2576 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 01 19:58:02.344765 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.344736 2576 fs.go:135] Filesystem UUIDs: map[28472917-cd5a-4041-9608-8bfe395becb0:/dev/nvme0n1p3 5ca2b1ae-7a09-4187-9f92-788c4705a8fd:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Nov 01 19:58:02.344832 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.344763 2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 01 19:58:02.345856 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.345841 2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 01 19:58:02.348994 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.348890 2576 manager.go:217] Machine: {Timestamp:2026-11-01 19:58:02.348423006 +0000 UTC m=+0.370149283 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3070390 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2059bf6e77f9d142f81e6481bc6c56 SystemUUID:ec2059bf-6e77-f9d1-42f8-1e6481bc6c56 BootID:47a34720-ea42-4c15-a637-08f630e1e3eb Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d5:f7:bb:6b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d5:f7:bb:6b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:da:f6:0f:61:46:b8 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 01 19:58:02.348994 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.348989 2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 01 19:58:02.349119 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.349059 2576 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 01 19:58:02.352997 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.352971 2576 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 01 19:58:02.353140 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.352998 2576 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-26.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 01 19:58:02.353191 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.353142 2576 topology_manager.go:143] "Creating topology manager with none policy" Nov 01 19:58:02.353191 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.353151 2576 container_manager_linux.go:311] "Creating device plugin manager" Nov 01 19:58:02.353191 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.353172 2576 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 19:58:02.353191 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.353187 2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 19:58:02.354352 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.354342 2576 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 01 19:58:02.358626 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.358613 2576 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 01 19:58:02.358743 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.358733 2576 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 01 19:58:02.358785 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.358764 2576 kubelet.go:493] "Attempting to sync node with API server" Nov 01 19:58:02.358785 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.358774 2576 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 01 19:58:02.358785 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.358783 2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 01 19:58:02.358898 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.358792 2576 kubelet.go:405] "Adding apiserver pod source" Nov 01 19:58:02.358898 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.358801 2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 01 19:58:02.359170 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.359155 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 19:58:02.361555 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.361539 2576 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 01 19:58:02.362713 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.362693 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-m7dj9" Nov 01 19:58:02.368874 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.368856 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 19:58:02.369832 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.369817 2576 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 01 19:58:02.369900 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.369857 2576 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 01 19:58:02.370045 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.370032 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 01 19:58:02.370121 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.370052 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 01 19:58:02.370121 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.370068 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 01 19:58:02.370121 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.370078 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 01 19:58:02.370121 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.370097 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 01 19:58:02.370121 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.370107 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 01 19:58:02.370444 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.370428 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-m7dj9" Nov 01 19:58:02.370823 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.370811 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 01 19:58:02.370929 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.370826 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 01 19:58:02.370929 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.370842 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 01 19:58:02.370929 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.370851 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 01 19:58:02.370929 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.370864 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 01 19:58:02.370929 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.370877 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 01 19:58:02.371708 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.371696 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 01 19:58:02.371768 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.371713 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 01 19:58:02.375697 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.375681 2576 server.go:1263] "Started kubelet" Nov 01 19:58:02.375781 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.375748 2576 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 01 19:58:02.375848 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.375759 2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 01 19:58:02.375901 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.375847 2576 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 01 19:58:02.376454 ip-10-0-136-26 systemd[1]: Started Kubernetes Kubelet. Nov 01 19:58:02.377872 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.377854 2576 server.go:317] "Adding debug handlers to kubelet server" Nov 01 19:58:02.377976 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.377964 2576 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 01 19:58:02.380432 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:02.379857 2576 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 01 19:58:02.383879 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.383860 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 01 19:58:02.384545 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.384527 2576 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 01 19:58:02.384793 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:02.384758 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-26.us-east-2.compute.internal\" not found" Nov 01 19:58:02.385357 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.385341 2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 01 19:58:02.385357 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.385340 2576 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 01 19:58:02.385471 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.385366 2576 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 01 19:58:02.385471 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.385432 2576 factory.go:55] Registering systemd factory Nov 01 19:58:02.385471 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.385445 2576 factory.go:223] Registration of the systemd container factory successfully Nov 01 19:58:02.385569 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.385519 2576 reconstruct.go:97] "Volume reconstruction finished" Nov 01 19:58:02.385569 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.385526 2576 reconciler.go:29] "Reconciler: start to sync state" Nov 01 19:58:02.385708 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.385669 2576 factory.go:153] Registering CRI-O factory Nov 01 19:58:02.385708 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.385688 2576 factory.go:223] Registration of the crio container factory successfully Nov 01 19:58:02.385793 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.385777 2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 01 19:58:02.385840 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.385797 2576 factory.go:103] Registering Raw factory Nov 01 19:58:02.385840 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.385833 2576 manager.go:1199] Started watching for new ooms in manager Nov 01 19:58:02.387000 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.386982 2576 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 19:58:02.387576 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.387561 2576 manager.go:319] Starting recovery of all containers Nov 01 19:58:02.388433 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.388407 2576 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-136-26.us-east-2.compute.internal" not found Nov 01 19:58:02.388520 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.388506 2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 19:58:02.395615 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.395593 2576 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 19:58:02.396330 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:02.396305 2576 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-136-26.us-east-2.compute.internal\" not found" node="ip-10-0-136-26.us-east-2.compute.internal" Nov 01 19:58:02.401877 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.401855 2576 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-136-26.us-east-2.compute.internal" not found Nov 01 19:58:02.407045 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.407025 2576 manager.go:324] Recovery completed Nov 01 19:58:02.411001 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.410988 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 19:58:02.412654 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.412637 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-26.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 19:58:02.412732 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.412672 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-26.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 19:58:02.412732 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.412688 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-26.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 19:58:02.413231 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.413217 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 19:58:02.413337 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.413218 2576 cpu_manager.go:226] "Starting" policy="none" Nov 01 19:58:02.413337 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.413247 2576 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 01 19:58:02.413337 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.413265 2576 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 01 19:58:02.413907 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.413893 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-26.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 19:58:02.413954 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.413923 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-26.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 19:58:02.413954 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.413935 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-26.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 19:58:02.415436 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.415423 2576 policy_none.go:50] "Start" Nov 01 19:58:02.415513 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.415440 2576 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 01 19:58:02.415513 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.415453 2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 01 19:58:02.417427 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.417414 2576 policy_none.go:44] "Start" Nov 01 19:58:02.453905 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.453871 2576 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 01 19:58:02.456986 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.456970 2576 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-136-26.us-east-2.compute.internal" not found Nov 01 19:58:02.458292 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.458270 2576 manager.go:341] "Starting Device Plugin manager" Nov 01 19:58:02.458362 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:02.458350 2576 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 01 19:58:02.458362 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.458363 2576 server.go:87] "Starting device plugin registration server" Nov 01 19:58:02.458617 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.458604 2576 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 01 19:58:02.458665 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.458620 2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 01 19:58:02.458725 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.458709 2576 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 01 19:58:02.458794 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.458774 2576 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 01 19:58:02.458794 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.458785 2576 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 01 19:58:02.459280 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:02.459253 2576 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 01 19:58:02.459385 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:02.459314 2576 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-26.us-east-2.compute.internal\" not found" Nov 01 19:58:02.536126 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.536104 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 01 19:58:02.538113 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.537241 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 01 19:58:02.538113 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.537261 2576 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 01 19:58:02.538113 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.537279 2576 kubelet.go:2521] "Starting kubelet main sync loop" Nov 01 19:58:02.538113 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:02.537327 2576 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 01 19:58:02.540851 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.540804 2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 19:58:02.559867 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.559853 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 19:58:02.560663 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.560643 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-26.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 19:58:02.560724 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.560671 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-26.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 19:58:02.560724 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.560685 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-26.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 19:58:02.560724 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.560713 2576 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-136-26.us-east-2.compute.internal" Nov 01 19:58:02.571510 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.571486 2576 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-136-26.us-east-2.compute.internal" Nov 01 19:58:02.638207 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.638186 2576 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-26.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-26.us-east-2.compute.internal"] Nov 01 19:58:02.640516 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.640502 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-26.us-east-2.compute.internal" Nov 01 19:58:02.642412 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.642393 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-26.us-east-2.compute.internal" Nov 01 19:58:02.673941 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.673924 2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-26.us-east-2.compute.internal" Nov 01 19:58:02.676373 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.676359 2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-26.us-east-2.compute.internal" Nov 01 19:58:02.686142 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.686118 2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 19:58:02.686844 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.686826 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d535588bb4025d791b22a311a0d6e9f5-config\") pod \"kube-apiserver-proxy-ip-10-0-136-26.us-east-2.compute.internal\" (UID: \"d535588bb4025d791b22a311a0d6e9f5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-26.us-east-2.compute.internal" Nov 01 19:58:02.686926 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.686855 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7f0999f9286775a451886a78f6ad439c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-26.us-east-2.compute.internal\" (UID: \"7f0999f9286775a451886a78f6ad439c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-26.us-east-2.compute.internal" Nov 01 19:58:02.686926 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.686880 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f0999f9286775a451886a78f6ad439c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-26.us-east-2.compute.internal\" (UID: \"7f0999f9286775a451886a78f6ad439c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-26.us-east-2.compute.internal" Nov 01 19:58:02.689173 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.689158 2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 19:58:02.787351 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.787313 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7f0999f9286775a451886a78f6ad439c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-26.us-east-2.compute.internal\" (UID: \"7f0999f9286775a451886a78f6ad439c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-26.us-east-2.compute.internal" Nov 01 19:58:02.787351 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.787353 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f0999f9286775a451886a78f6ad439c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-26.us-east-2.compute.internal\" (UID: \"7f0999f9286775a451886a78f6ad439c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-26.us-east-2.compute.internal" Nov 01 19:58:02.787503 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.787372 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d535588bb4025d791b22a311a0d6e9f5-config\") pod \"kube-apiserver-proxy-ip-10-0-136-26.us-east-2.compute.internal\" (UID: \"d535588bb4025d791b22a311a0d6e9f5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-26.us-east-2.compute.internal" Nov 01 19:58:02.787503 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.787420 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d535588bb4025d791b22a311a0d6e9f5-config\") pod \"kube-apiserver-proxy-ip-10-0-136-26.us-east-2.compute.internal\" (UID: \"d535588bb4025d791b22a311a0d6e9f5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-26.us-east-2.compute.internal" Nov 01 19:58:02.787503 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.787429 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7f0999f9286775a451886a78f6ad439c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-26.us-east-2.compute.internal\" (UID: \"7f0999f9286775a451886a78f6ad439c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-26.us-east-2.compute.internal" Nov 01 19:58:02.787503 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.787429 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f0999f9286775a451886a78f6ad439c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-26.us-east-2.compute.internal\" (UID: \"7f0999f9286775a451886a78f6ad439c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-26.us-east-2.compute.internal" Nov 01 19:58:02.990636 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.990607 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-26.us-east-2.compute.internal" Nov 01 19:58:02.992767 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:02.992750 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-26.us-east-2.compute.internal" Nov 01 19:58:03.313365 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.313157 2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 01 19:58:03.313722 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.313471 2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 01 19:58:03.313722 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.313471 2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 01 19:58:03.313722 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.313489 2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 01 19:58:03.359931 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.359914 2576 apiserver.go:52] "Watching apiserver" Nov 01 19:58:03.366710 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.366695 2576 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 01 19:58:03.366962 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.366947 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-cffbl","openshift-cluster-node-tuning-operator/tuned-r9cvk","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-26.us-east-2.compute.internal","openshift-multus/multus-588fk","kube-system/konnectivity-agent-rz7h2","kube-system/kube-apiserver-proxy-ip-10-0-136-26.us-east-2.compute.internal","openshift-multus/multus-additional-cni-plugins-bltk8","openshift-multus/network-metrics-daemon-bvqvw","kube-system/global-pull-secret-syncer-k5brd","openshift-image-registry/node-ca-77fcl","openshift-network-diagnostics/network-check-target-b7mcz","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvhjt","openshift-dns/node-resolver-tzgt2","openshift-network-operator/iptables-alerter-dg5c2"] Nov 01 19:58:03.369098 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.369081 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:58:03.370857 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.370842 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:58:03.372651 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.372633 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-588fk" Nov 01 19:58:03.372764 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.372699 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 01 19:58:03.372764 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.372715 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 01 19:58:03.373676 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.373659 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 01 19:58:03.373738 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.373712 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 01 19:58:03.373738 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.373724 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 01 19:58:03.373802 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.373769 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-754ws\"" Nov 01 19:58:03.373984 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.373972 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 01 19:58:03.374038 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.373974 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 01 19:58:03.374430 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.374413 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rz7h2" Nov 01 19:58:03.374956 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.374942 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-tdbxs\"" Nov 01 19:58:03.375049 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.375036 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 01 19:58:03.375111 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.375102 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 01 19:58:03.375190 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.375178 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 01 19:58:03.375234 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.375182 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 01 19:58:03.375234 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.375182 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-24gff\"" Nov 01 19:58:03.375407 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.375396 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 01 19:58:03.375833 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.375795 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 19:53:02 +0000 UTC" deadline="2028-05-07 20:49:58.134929996 +0000 UTC" Nov 01 19:58:03.375866 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.375835 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13272h51m54.759098593s" Nov 01 19:58:03.376334 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.376320 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bltk8" Nov 01 19:58:03.378924 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.376959 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 01 19:58:03.378924 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.377025 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 01 19:58:03.378924 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.377184 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-j6fxj\"" Nov 01 19:58:03.378924 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.378674 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 01 19:58:03.378924 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.378860 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 01 19:58:03.378924 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.378879 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-n6s44\"" Nov 01 19:58:03.379169 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.378934 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bvqvw" Nov 01 19:58:03.379169 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:03.379007 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bvqvw" podUID="65e6a200-c6f4-4671-9d2e-b0158ff3c5ce" Nov 01 19:58:03.380740 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.380726 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k5brd" Nov 01 19:58:03.380821 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:03.380788 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k5brd" podUID="2efc5bbf-efd4-4a42-bed8-53f2490bb7bc" Nov 01 19:58:03.382508 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.382493 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-77fcl" Nov 01 19:58:03.384065 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.384051 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 01 19:58:03.384322 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.384308 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b7mcz" Nov 01 19:58:03.384396 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:03.384354 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b7mcz" podUID="16e51d04-90d5-484d-9f85-e9d484205220" Nov 01 19:58:03.386293 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.386263 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvhjt" Nov 01 19:58:03.388318 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.388303 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tzgt2" Nov 01 19:58:03.390316 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.390301 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dg5c2" Nov 01 19:58:03.391493 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.391477 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/91ef232e-99ae-42f2-be58-e17cc42f12ea-device-dir\") pod \"aws-ebs-csi-driver-node-fvhjt\" (UID: \"91ef232e-99ae-42f2-be58-e17cc42f12ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvhjt" Nov 01 19:58:03.391550 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.391501 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5a57adad-0c11-4902-be16-b8dfb4e3ee13-lib-modules\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:58:03.391550 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.391516 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8537e78c-5b2c-4966-9558-044ddc39119b-konnectivity-ca\") pod \"konnectivity-agent-rz7h2\" (UID: \"8537e78c-5b2c-4966-9558-044ddc39119b\") " pod="kube-system/konnectivity-agent-rz7h2" Nov 01 19:58:03.391550 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.391536 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq4k2\" (UniqueName: \"kubernetes.io/projected/16e51d04-90d5-484d-9f85-e9d484205220-kube-api-access-vq4k2\") pod \"network-check-target-b7mcz\" (UID: \"16e51d04-90d5-484d-9f85-e9d484205220\") " pod="openshift-network-diagnostics/network-check-target-b7mcz" Nov 01 19:58:03.391666 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.391557 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a57adad-0c11-4902-be16-b8dfb4e3ee13-host\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:58:03.391666 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.391599 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-cnibin\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:58:03.391666 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.391638 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9e564268-fa85-48e6-8d1c-5069cc0f9648-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bltk8\" (UID: \"9e564268-fa85-48e6-8d1c-5069cc0f9648\") " pod="openshift-multus/multus-additional-cni-plugins-bltk8" Nov 01 19:58:03.391751 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.391678 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2efc5bbf-efd4-4a42-bed8-53f2490bb7bc-kubelet-config\") pod \"global-pull-secret-syncer-k5brd\" (UID: \"2efc5bbf-efd4-4a42-bed8-53f2490bb7bc\") " pod="kube-system/global-pull-secret-syncer-k5brd" Nov 01 19:58:03.391751 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.391711 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fntm\" (UniqueName: \"kubernetes.io/projected/983a921d-61a5-4668-8e17-1a24b6eaf8b6-kube-api-access-8fntm\") pod \"node-ca-77fcl\" (UID: \"983a921d-61a5-4668-8e17-1a24b6eaf8b6\") " pod="openshift-image-registry/node-ca-77fcl" Nov 01 19:58:03.391751 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.391739 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c0e3a247-a792-431d-85cb-9b46982c3373-hosts-file\") pod \"node-resolver-tzgt2\" (UID: \"c0e3a247-a792-431d-85cb-9b46982c3373\") " pod="openshift-dns/node-resolver-tzgt2" Nov 01 19:58:03.391836 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.391762 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5a57adad-0c11-4902-be16-b8dfb4e3ee13-etc-tuned\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:58:03.391836 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.391775 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-host-var-lib-cni-bin\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:58:03.391836 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.391791 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9e564268-fa85-48e6-8d1c-5069cc0f9648-system-cni-dir\") pod \"multus-additional-cni-plugins-bltk8\" (UID: \"9e564268-fa85-48e6-8d1c-5069cc0f9648\") " pod="openshift-multus/multus-additional-cni-plugins-bltk8" Nov 01 19:58:03.391836 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.391819 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/983a921d-61a5-4668-8e17-1a24b6eaf8b6-host\") pod \"node-ca-77fcl\" (UID: \"983a921d-61a5-4668-8e17-1a24b6eaf8b6\") " pod="openshift-image-registry/node-ca-77fcl" Nov 01 19:58:03.391946 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.391840 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-run-systemd\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:58:03.391946 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.391867 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/91ef232e-99ae-42f2-be58-e17cc42f12ea-etc-selinux\") pod \"aws-ebs-csi-driver-node-fvhjt\" (UID: \"91ef232e-99ae-42f2-be58-e17cc42f12ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvhjt" Nov 01 19:58:03.391946 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.391884 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:58:03.391946 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.391909 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-host-run-multus-certs\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:58:03.392084 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.391944 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2efc5bbf-efd4-4a42-bed8-53f2490bb7bc-original-pull-secret\") pod \"global-pull-secret-syncer-k5brd\" (UID: \"2efc5bbf-efd4-4a42-bed8-53f2490bb7bc\") " pod="kube-system/global-pull-secret-syncer-k5brd" Nov 01 19:58:03.392084 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.391960 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-host-slash\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:58:03.392084 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.391977 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-host-cni-netd\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:58:03.392084 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.391997 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz4mx\" (UniqueName: \"kubernetes.io/projected/c0e3a247-a792-431d-85cb-9b46982c3373-kube-api-access-nz4mx\") pod \"node-resolver-tzgt2\" (UID: \"c0e3a247-a792-431d-85cb-9b46982c3373\") " pod="openshift-dns/node-resolver-tzgt2" Nov 01 19:58:03.392084 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.392043 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5a57adad-0c11-4902-be16-b8dfb4e3ee13-etc-sysctl-d\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:58:03.392084 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.392071 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a57adad-0c11-4902-be16-b8dfb4e3ee13-tmp\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:58:03.392362 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.392094 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhv76\" (UniqueName: \"kubernetes.io/projected/62c9da55-b21d-4f8f-a3ec-98e823655351-kube-api-access-fhv76\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:58:03.392362 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.392144 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9e564268-fa85-48e6-8d1c-5069cc0f9648-cni-binary-copy\") pod \"multus-additional-cni-plugins-bltk8\" (UID: \"9e564268-fa85-48e6-8d1c-5069cc0f9648\") " pod="openshift-multus/multus-additional-cni-plugins-bltk8" Nov 01 19:58:03.392362 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.392171 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-host-run-netns\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:58:03.392362 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.392194 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-ovn-node-metrics-cert\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:58:03.392362 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.392218 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0e3a247-a792-431d-85cb-9b46982c3373-tmp-dir\") pod \"node-resolver-tzgt2\" (UID: \"c0e3a247-a792-431d-85cb-9b46982c3373\") " pod="openshift-dns/node-resolver-tzgt2" Nov 01 19:58:03.392362 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.392252 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5a57adad-0c11-4902-be16-b8dfb4e3ee13-etc-sysctl-conf\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:58:03.392362 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.392313 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a57adad-0c11-4902-be16-b8dfb4e3ee13-etc-kubernetes\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:58:03.392362 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.392332 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-multus-socket-dir-parent\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:58:03.392362 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.392349 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-host-var-lib-cni-multus\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:58:03.392756 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.392392 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-hostroot\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:58:03.392756 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.392421 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-run-ovn\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:58:03.392756 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.392453 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-ovnkube-script-lib\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:58:03.392756 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.392513 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98mhk\" (UniqueName: \"kubernetes.io/projected/5a57adad-0c11-4902-be16-b8dfb4e3ee13-kube-api-access-98mhk\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:58:03.392756 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.392536 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-host-run-netns\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:58:03.392756 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.392556 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-etc-kubernetes\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:58:03.392756 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.392576 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5a57adad-0c11-4902-be16-b8dfb4e3ee13-etc-systemd\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:58:03.392756 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.392596 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a57adad-0c11-4902-be16-b8dfb4e3ee13-var-lib-kubelet\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:58:03.392756 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.392622 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/62c9da55-b21d-4f8f-a3ec-98e823655351-multus-daemon-config\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:58:03.392756 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.392659 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-node-log\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:58:03.392756 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.392689 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/91ef232e-99ae-42f2-be58-e17cc42f12ea-socket-dir\") pod \"aws-ebs-csi-driver-node-fvhjt\" (UID: \"91ef232e-99ae-42f2-be58-e17cc42f12ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvhjt" Nov 01 19:58:03.392756 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.392724 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5a57adad-0c11-4902-be16-b8dfb4e3ee13-run\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:58:03.392756 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.392750 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9e564268-fa85-48e6-8d1c-5069cc0f9648-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bltk8\" (UID: \"9e564268-fa85-48e6-8d1c-5069cc0f9648\") " pod="openshift-multus/multus-additional-cni-plugins-bltk8" Nov 01 19:58:03.393254 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.392784 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2efc5bbf-efd4-4a42-bed8-53f2490bb7bc-dbus\") pod \"global-pull-secret-syncer-k5brd\" (UID: \"2efc5bbf-efd4-4a42-bed8-53f2490bb7bc\") " pod="kube-system/global-pull-secret-syncer-k5brd" Nov 01 19:58:03.393254 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.392809 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-var-lib-openvswitch\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:58:03.393254 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.392841 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-host-cni-bin\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:58:03.393254 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.392865 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-ovnkube-config\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:58:03.393254 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.392886 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-system-cni-dir\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:58:03.393254 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.392911 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-host-kubelet\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:58:03.393254 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.392932 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-run-openvswitch\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:58:03.393254 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.392961 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-multus-conf-dir\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:58:03.393254 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.392990 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9e564268-fa85-48e6-8d1c-5069cc0f9648-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bltk8\" (UID: \"9e564268-fa85-48e6-8d1c-5069cc0f9648\") " pod="openshift-multus/multus-additional-cni-plugins-bltk8" Nov 01 19:58:03.393254 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.393016 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnjs4\" (UniqueName: \"kubernetes.io/projected/9e564268-fa85-48e6-8d1c-5069cc0f9648-kube-api-access-mnjs4\") pod \"multus-additional-cni-plugins-bltk8\" (UID: \"9e564268-fa85-48e6-8d1c-5069cc0f9648\") " pod="openshift-multus/multus-additional-cni-plugins-bltk8" Nov 01 19:58:03.393254 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.393051 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-host-run-ovn-kubernetes\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:58:03.393254 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.393080 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/91ef232e-99ae-42f2-be58-e17cc42f12ea-sys-fs\") pod \"aws-ebs-csi-driver-node-fvhjt\" (UID: \"91ef232e-99ae-42f2-be58-e17cc42f12ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvhjt" Nov 01 19:58:03.393254 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.393099 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5a57adad-0c11-4902-be16-b8dfb4e3ee13-etc-sysconfig\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:58:03.393254 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.393164 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-multus-cni-dir\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:58:03.393254 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.393182 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/62c9da55-b21d-4f8f-a3ec-98e823655351-cni-binary-copy\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:58:03.393254 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.393194 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-systemd-units\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:58:03.393954 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.393206 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-log-socket\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:58:03.393954 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.393218 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/91ef232e-99ae-42f2-be58-e17cc42f12ea-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fvhjt\" (UID: \"91ef232e-99ae-42f2-be58-e17cc42f12ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvhjt" Nov 01 19:58:03.393954 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.393235 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9e564268-fa85-48e6-8d1c-5069cc0f9648-cnibin\") pod \"multus-additional-cni-plugins-bltk8\" (UID: \"9e564268-fa85-48e6-8d1c-5069cc0f9648\") " pod="openshift-multus/multus-additional-cni-plugins-bltk8" Nov 01 19:58:03.393954 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.393275 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtnhd\" (UniqueName: \"kubernetes.io/projected/65e6a200-c6f4-4671-9d2e-b0158ff3c5ce-kube-api-access-jtnhd\") pod \"network-metrics-daemon-bvqvw\" (UID: \"65e6a200-c6f4-4671-9d2e-b0158ff3c5ce\") " pod="openshift-multus/network-metrics-daemon-bvqvw" Nov 01 19:58:03.393954 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.393324 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-etc-openvswitch\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:58:03.393954 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.393350 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-env-overrides\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:58:03.393954 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.393367 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/91ef232e-99ae-42f2-be58-e17cc42f12ea-registration-dir\") pod \"aws-ebs-csi-driver-node-fvhjt\" (UID: \"91ef232e-99ae-42f2-be58-e17cc42f12ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvhjt" Nov 01 19:58:03.393954 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.393401 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z6z7\" (UniqueName: \"kubernetes.io/projected/91ef232e-99ae-42f2-be58-e17cc42f12ea-kube-api-access-8z6z7\") pod \"aws-ebs-csi-driver-node-fvhjt\" (UID: \"91ef232e-99ae-42f2-be58-e17cc42f12ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvhjt" Nov 01 19:58:03.393954 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.393432 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5a57adad-0c11-4902-be16-b8dfb4e3ee13-etc-modprobe-d\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:58:03.393954 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.393454 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a57adad-0c11-4902-be16-b8dfb4e3ee13-sys\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:58:03.393954 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.393468 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-host-var-lib-kubelet\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:58:03.393954 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.393490 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/983a921d-61a5-4668-8e17-1a24b6eaf8b6-serviceca\") pod \"node-ca-77fcl\" (UID: \"983a921d-61a5-4668-8e17-1a24b6eaf8b6\") " pod="openshift-image-registry/node-ca-77fcl" Nov 01 19:58:03.393954 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.393503 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dg6b\" (UniqueName: \"kubernetes.io/projected/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-kube-api-access-6dg6b\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:58:03.393954 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.393568 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8537e78c-5b2c-4966-9558-044ddc39119b-agent-certs\") pod \"konnectivity-agent-rz7h2\" (UID: \"8537e78c-5b2c-4966-9558-044ddc39119b\") " pod="kube-system/konnectivity-agent-rz7h2" Nov 01 19:58:03.393954 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.393582 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65e6a200-c6f4-4671-9d2e-b0158ff3c5ce-metrics-certs\") pod \"network-metrics-daemon-bvqvw\" (UID: \"65e6a200-c6f4-4671-9d2e-b0158ff3c5ce\") " pod="openshift-multus/network-metrics-daemon-bvqvw" Nov 01 19:58:03.393954 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.393596 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-host-run-k8s-cni-cncf-io\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:58:03.396945 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.396908 2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 01 19:58:03.408441 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.408425 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-nccls" Nov 01 19:58:03.424194 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.424179 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-nccls" Nov 01 19:58:03.464748 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.464718 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 01 19:58:03.465024 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.464826 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 01 19:58:03.465024 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.464843 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 01 19:58:03.465024 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.464849 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 01 19:58:03.465024 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.464871 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 01 19:58:03.465024 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.464888 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rbpdw\"" Nov 01 19:58:03.465024 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.464971 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 01 19:58:03.465024 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.465002 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 01 19:58:03.465203 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.465064 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zgwx5\"" Nov 01 19:58:03.465203 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.465088 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 01 19:58:03.465788 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.465762 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 01 19:58:03.465788 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.465765 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 01 19:58:03.465788 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.465784 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 01 19:58:03.465950 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.465801 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5qrts\"" Nov 01 19:58:03.465950 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.465845 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-f8578\"" Nov 01 19:58:03.486029 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.486012 2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 01 19:58:03.493819 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.493804 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/62c9da55-b21d-4f8f-a3ec-98e823655351-multus-daemon-config\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:58:03.493873 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.493827 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-node-log\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:58:03.493873 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.493841 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/91ef232e-99ae-42f2-be58-e17cc42f12ea-socket-dir\") pod \"aws-ebs-csi-driver-node-fvhjt\" (UID: \"91ef232e-99ae-42f2-be58-e17cc42f12ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvhjt" Nov 01 19:58:03.493873 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.493860 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5a57adad-0c11-4902-be16-b8dfb4e3ee13-run\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:58:03.494042 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.493879 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9e564268-fa85-48e6-8d1c-5069cc0f9648-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bltk8\" (UID: \"9e564268-fa85-48e6-8d1c-5069cc0f9648\") " pod="openshift-multus/multus-additional-cni-plugins-bltk8" Nov 01 19:58:03.494042 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.493899 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-node-log\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:58:03.494042 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.493919 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2efc5bbf-efd4-4a42-bed8-53f2490bb7bc-dbus\") pod \"global-pull-secret-syncer-k5brd\" (UID: \"2efc5bbf-efd4-4a42-bed8-53f2490bb7bc\") " pod="kube-system/global-pull-secret-syncer-k5brd" Nov 01 19:58:03.494042 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.493953 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-var-lib-openvswitch\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:58:03.494042 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.493974 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-host-cni-bin\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:58:03.494042 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.494001 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5a57adad-0c11-4902-be16-b8dfb4e3ee13-run\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:58:03.494042 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.494012 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2efc5bbf-efd4-4a42-bed8-53f2490bb7bc-dbus\") pod \"global-pull-secret-syncer-k5brd\" (UID: \"2efc5bbf-efd4-4a42-bed8-53f2490bb7bc\") " pod="kube-system/global-pull-secret-syncer-k5brd" Nov 01 19:58:03.494042 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.494006 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9xlj\" (UniqueName: \"kubernetes.io/projected/f16f57a8-9630-416c-bd95-f5a2c6af2f15-kube-api-access-j9xlj\") pod \"iptables-alerter-dg5c2\" (UID: \"f16f57a8-9630-416c-bd95-f5a2c6af2f15\") " pod="openshift-network-operator/iptables-alerter-dg5c2" Nov 01 19:58:03.494042 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.494017 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-var-lib-openvswitch\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:58:03.494483 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.494066 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-host-cni-bin\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:58:03.494483 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.494080 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-ovnkube-config\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:58:03.494483 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.494103 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-system-cni-dir\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:58:03.494483 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.494124 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-host-kubelet\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:58:03.494483 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.494145 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-run-openvswitch\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:58:03.494483 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.494167 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-multus-conf-dir\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:58:03.494483 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.494176 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-host-kubelet\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:58:03.494483 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.494191 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9e564268-fa85-48e6-8d1c-5069cc0f9648-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bltk8\" (UID: \"9e564268-fa85-48e6-8d1c-5069cc0f9648\") " pod="openshift-multus/multus-additional-cni-plugins-bltk8" Nov 01 19:58:03.494483 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.494201 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-run-openvswitch\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:58:03.494483 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.494215 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnjs4\" (UniqueName: \"kubernetes.io/projected/9e564268-fa85-48e6-8d1c-5069cc0f9648-kube-api-access-mnjs4\") pod \"multus-additional-cni-plugins-bltk8\" (UID: \"9e564268-fa85-48e6-8d1c-5069cc0f9648\") " pod="openshift-multus/multus-additional-cni-plugins-bltk8" Nov 01 19:58:03.494483 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.494226 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-multus-conf-dir\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:58:03.494483 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.494226 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-system-cni-dir\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:58:03.494483 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.494236 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-host-run-ovn-kubernetes\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:58:03.494483 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.494260 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/91ef232e-99ae-42f2-be58-e17cc42f12ea-sys-fs\") pod \"aws-ebs-csi-driver-node-fvhjt\" (UID: \"91ef232e-99ae-42f2-be58-e17cc42f12ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvhjt" Nov 01 19:58:03.494483 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.494300 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5a57adad-0c11-4902-be16-b8dfb4e3ee13-etc-sysconfig\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:58:03.494483 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.494306 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-host-run-ovn-kubernetes\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:58:03.494483 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.494325 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-multus-cni-dir\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:58:03.495214 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.494343 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/62c9da55-b21d-4f8f-a3ec-98e823655351-cni-binary-copy\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:58:03.495214 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.494357 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5a57adad-0c11-4902-be16-b8dfb4e3ee13-etc-sysconfig\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:58:03.495214 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.494362 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-systemd-units\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:58:03.495214 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.494383 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-log-socket\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:58:03.495214 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.494444 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-log-socket\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:58:03.495214 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.494443 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/91ef232e-99ae-42f2-be58-e17cc42f12ea-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fvhjt\" (UID: \"91ef232e-99ae-42f2-be58-e17cc42f12ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvhjt" Nov 01 19:58:03.495214 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.494466 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/91ef232e-99ae-42f2-be58-e17cc42f12ea-socket-dir\") pod \"aws-ebs-csi-driver-node-fvhjt\" (UID: \"91ef232e-99ae-42f2-be58-e17cc42f12ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvhjt" Nov 01 19:58:03.495214 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.494479 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9e564268-fa85-48e6-8d1c-5069cc0f9648-cnibin\") pod \"multus-additional-cni-plugins-bltk8\" (UID: \"9e564268-fa85-48e6-8d1c-5069cc0f9648\") " pod="openshift-multus/multus-additional-cni-plugins-bltk8" Nov 01 19:58:03.495214 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.494483 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/91ef232e-99ae-42f2-be58-e17cc42f12ea-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fvhjt\" (UID: \"91ef232e-99ae-42f2-be58-e17cc42f12ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvhjt" Nov 01 19:58:03.495214 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.494507 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtnhd\" (UniqueName: \"kubernetes.io/projected/65e6a200-c6f4-4671-9d2e-b0158ff3c5ce-kube-api-access-jtnhd\") pod \"network-metrics-daemon-bvqvw\" (UID: \"65e6a200-c6f4-4671-9d2e-b0158ff3c5ce\") " pod="openshift-multus/network-metrics-daemon-bvqvw" Nov 01 19:58:03.495214 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.494525 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9e564268-fa85-48e6-8d1c-5069cc0f9648-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bltk8\" (UID: \"9e564268-fa85-48e6-8d1c-5069cc0f9648\") " pod="openshift-multus/multus-additional-cni-plugins-bltk8" Nov 01 19:58:03.495214 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.494527 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/91ef232e-99ae-42f2-be58-e17cc42f12ea-sys-fs\") pod \"aws-ebs-csi-driver-node-fvhjt\" (UID: \"91ef232e-99ae-42f2-be58-e17cc42f12ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvhjt" Nov 01 19:58:03.495214 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.494542 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-systemd-units\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:58:03.495214 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.494563 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-etc-openvswitch\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:58:03.495214 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.494579 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-multus-cni-dir\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:58:03.495214 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.494585 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-env-overrides\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:58:03.495214 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.494620 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9e564268-fa85-48e6-8d1c-5069cc0f9648-cnibin\") pod \"multus-additional-cni-plugins-bltk8\" (UID: \"9e564268-fa85-48e6-8d1c-5069cc0f9648\") " pod="openshift-multus/multus-additional-cni-plugins-bltk8" Nov 01 19:58:03.495941 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.494662 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/91ef232e-99ae-42f2-be58-e17cc42f12ea-registration-dir\") pod \"aws-ebs-csi-driver-node-fvhjt\" (UID: \"91ef232e-99ae-42f2-be58-e17cc42f12ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvhjt" Nov 01 19:58:03.495941 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.494695 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z6z7\" (UniqueName: \"kubernetes.io/projected/91ef232e-99ae-42f2-be58-e17cc42f12ea-kube-api-access-8z6z7\") pod \"aws-ebs-csi-driver-node-fvhjt\" (UID: \"91ef232e-99ae-42f2-be58-e17cc42f12ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvhjt" Nov 01 19:58:03.495941 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.494718 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5a57adad-0c11-4902-be16-b8dfb4e3ee13-etc-modprobe-d\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:58:03.495941 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.494741 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a57adad-0c11-4902-be16-b8dfb4e3ee13-sys\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:58:03.495941 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.494789 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-host-var-lib-kubelet\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:58:03.495941 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.494816 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/983a921d-61a5-4668-8e17-1a24b6eaf8b6-serviceca\") pod \"node-ca-77fcl\" (UID: \"983a921d-61a5-4668-8e17-1a24b6eaf8b6\") " pod="openshift-image-registry/node-ca-77fcl" Nov 01 19:58:03.495941 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.494840 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dg6b\" (UniqueName: \"kubernetes.io/projected/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-kube-api-access-6dg6b\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:58:03.495941 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.494862 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8537e78c-5b2c-4966-9558-044ddc39119b-agent-certs\") pod \"konnectivity-agent-rz7h2\" (UID: \"8537e78c-5b2c-4966-9558-044ddc39119b\") " pod="kube-system/konnectivity-agent-rz7h2" Nov 01 19:58:03.495941 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.494873 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9e564268-fa85-48e6-8d1c-5069cc0f9648-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bltk8\" (UID: \"9e564268-fa85-48e6-8d1c-5069cc0f9648\") " pod="openshift-multus/multus-additional-cni-plugins-bltk8" Nov 01 19:58:03.495941 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.494885 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65e6a200-c6f4-4671-9d2e-b0158ff3c5ce-metrics-certs\") pod \"network-metrics-daemon-bvqvw\" (UID: \"65e6a200-c6f4-4671-9d2e-b0158ff3c5ce\") " pod="openshift-multus/network-metrics-daemon-bvqvw" Nov 01 19:58:03.495941 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.494926 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-host-var-lib-kubelet\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:58:03.495941 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.494926 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-host-run-k8s-cni-cncf-io\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:58:03.495941 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.494956 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/91ef232e-99ae-42f2-be58-e17cc42f12ea-device-dir\") pod \"aws-ebs-csi-driver-node-fvhjt\" (UID: \"91ef232e-99ae-42f2-be58-e17cc42f12ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvhjt" Nov 01 19:58:03.495941 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.494965 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-host-run-k8s-cni-cncf-io\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:58:03.495941 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.494980 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5a57adad-0c11-4902-be16-b8dfb4e3ee13-lib-modules\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:58:03.495941 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:03.495068 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 19:58:03.495941 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.495076 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a57adad-0c11-4902-be16-b8dfb4e3ee13-sys\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:58:03.495941 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.495103 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8537e78c-5b2c-4966-9558-044ddc39119b-konnectivity-ca\") pod \"konnectivity-agent-rz7h2\" (UID: \"8537e78c-5b2c-4966-9558-044ddc39119b\") " pod="kube-system/konnectivity-agent-rz7h2" Nov 01 19:58:03.496804 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:03.495116 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65e6a200-c6f4-4671-9d2e-b0158ff3c5ce-metrics-certs podName:65e6a200-c6f4-4671-9d2e-b0158ff3c5ce nodeName:}" failed. No retries permitted until 2026-11-01 19:58:03.995100361 +0000 UTC m=+2.016826638 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/65e6a200-c6f4-4671-9d2e-b0158ff3c5ce-metrics-certs") pod "network-metrics-daemon-bvqvw" (UID: "65e6a200-c6f4-4671-9d2e-b0158ff3c5ce") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 19:58:03.496804 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.495137 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/62c9da55-b21d-4f8f-a3ec-98e823655351-multus-daemon-config\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:58:03.496804 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.495143 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq4k2\" (UniqueName: \"kubernetes.io/projected/16e51d04-90d5-484d-9f85-e9d484205220-kube-api-access-vq4k2\") pod \"network-check-target-b7mcz\" (UID: \"16e51d04-90d5-484d-9f85-e9d484205220\") " pod="openshift-network-diagnostics/network-check-target-b7mcz" Nov 01 19:58:03.496804 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.495169 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a57adad-0c11-4902-be16-b8dfb4e3ee13-host\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:58:03.496804 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.495188 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-etc-openvswitch\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:58:03.496804 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.495194 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f16f57a8-9630-416c-bd95-f5a2c6af2f15-host-slash\") pod \"iptables-alerter-dg5c2\" (UID: \"f16f57a8-9630-416c-bd95-f5a2c6af2f15\") " pod="openshift-network-operator/iptables-alerter-dg5c2" Nov 01 19:58:03.496804 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.495241 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/91ef232e-99ae-42f2-be58-e17cc42f12ea-registration-dir\") pod \"aws-ebs-csi-driver-node-fvhjt\" (UID: \"91ef232e-99ae-42f2-be58-e17cc42f12ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvhjt" Nov 01 19:58:03.496804 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.495298 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-cnibin\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:58:03.496804 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.495325 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9e564268-fa85-48e6-8d1c-5069cc0f9648-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bltk8\" (UID: \"9e564268-fa85-48e6-8d1c-5069cc0f9648\") " pod="openshift-multus/multus-additional-cni-plugins-bltk8" Nov 01 19:58:03.496804 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.495349 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2efc5bbf-efd4-4a42-bed8-53f2490bb7bc-kubelet-config\") pod \"global-pull-secret-syncer-k5brd\" (UID: \"2efc5bbf-efd4-4a42-bed8-53f2490bb7bc\") " pod="kube-system/global-pull-secret-syncer-k5brd" Nov 01 19:58:03.496804 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.495352 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-ovnkube-config\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:58:03.496804 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.495372 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fntm\" (UniqueName: \"kubernetes.io/projected/983a921d-61a5-4668-8e17-1a24b6eaf8b6-kube-api-access-8fntm\") pod \"node-ca-77fcl\" (UID: \"983a921d-61a5-4668-8e17-1a24b6eaf8b6\") " pod="openshift-image-registry/node-ca-77fcl" Nov 01 19:58:03.496804 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.495429 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c0e3a247-a792-431d-85cb-9b46982c3373-hosts-file\") pod \"node-resolver-tzgt2\" (UID: \"c0e3a247-a792-431d-85cb-9b46982c3373\") " pod="openshift-dns/node-resolver-tzgt2" Nov 01 19:58:03.496804 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.495447 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/983a921d-61a5-4668-8e17-1a24b6eaf8b6-serviceca\") pod \"node-ca-77fcl\" (UID: \"983a921d-61a5-4668-8e17-1a24b6eaf8b6\") " pod="openshift-image-registry/node-ca-77fcl" Nov 01 19:58:03.496804 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.495453 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5a57adad-0c11-4902-be16-b8dfb4e3ee13-etc-tuned\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:58:03.496804 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.495476 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-host-var-lib-cni-bin\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:58:03.496804 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.495500 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/91ef232e-99ae-42f2-be58-e17cc42f12ea-device-dir\") pod \"aws-ebs-csi-driver-node-fvhjt\" (UID: \"91ef232e-99ae-42f2-be58-e17cc42f12ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvhjt" Nov 01 19:58:03.497824 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.495514 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9e564268-fa85-48e6-8d1c-5069cc0f9648-system-cni-dir\") pod \"multus-additional-cni-plugins-bltk8\" (UID: \"9e564268-fa85-48e6-8d1c-5069cc0f9648\") " pod="openshift-multus/multus-additional-cni-plugins-bltk8" Nov 01 19:58:03.497824 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.495570 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8537e78c-5b2c-4966-9558-044ddc39119b-konnectivity-ca\") pod \"konnectivity-agent-rz7h2\" (UID: \"8537e78c-5b2c-4966-9558-044ddc39119b\") " pod="kube-system/konnectivity-agent-rz7h2" Nov 01 19:58:03.497824 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.495611 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a57adad-0c11-4902-be16-b8dfb4e3ee13-host\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:58:03.497824 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.495620 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2efc5bbf-efd4-4a42-bed8-53f2490bb7bc-kubelet-config\") pod \"global-pull-secret-syncer-k5brd\" (UID: \"2efc5bbf-efd4-4a42-bed8-53f2490bb7bc\") " pod="kube-system/global-pull-secret-syncer-k5brd" Nov 01 19:58:03.497824 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.495632 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5a57adad-0c11-4902-be16-b8dfb4e3ee13-lib-modules\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:58:03.497824 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.495657 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-cnibin\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:58:03.497824 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.495734 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c0e3a247-a792-431d-85cb-9b46982c3373-hosts-file\") pod \"node-resolver-tzgt2\" (UID: \"c0e3a247-a792-431d-85cb-9b46982c3373\") " pod="openshift-dns/node-resolver-tzgt2" Nov 01 19:58:03.497824 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.495742 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5a57adad-0c11-4902-be16-b8dfb4e3ee13-etc-modprobe-d\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:58:03.497824 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.495758 2576 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 01 19:58:03.497824 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.495783 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/983a921d-61a5-4668-8e17-1a24b6eaf8b6-host\") pod \"node-ca-77fcl\" (UID: \"983a921d-61a5-4668-8e17-1a24b6eaf8b6\") " pod="openshift-image-registry/node-ca-77fcl" Nov 01 19:58:03.497824 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.495810 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-run-systemd\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:58:03.497824 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.495814 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-host-var-lib-cni-bin\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:58:03.497824 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.495854 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/91ef232e-99ae-42f2-be58-e17cc42f12ea-etc-selinux\") pod \"aws-ebs-csi-driver-node-fvhjt\" (UID: \"91ef232e-99ae-42f2-be58-e17cc42f12ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvhjt" Nov 01 19:58:03.497824 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.495878 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:58:03.497824 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.495904 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-host-run-multus-certs\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:58:03.497824 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.495969 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2efc5bbf-efd4-4a42-bed8-53f2490bb7bc-original-pull-secret\") pod \"global-pull-secret-syncer-k5brd\" (UID: \"2efc5bbf-efd4-4a42-bed8-53f2490bb7bc\") " pod="kube-system/global-pull-secret-syncer-k5brd" Nov 01 19:58:03.497824 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.496019 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-host-slash\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:58:03.497824 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.496038 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-host-cni-netd\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:58:03.498703 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.496057 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz4mx\" (UniqueName: \"kubernetes.io/projected/c0e3a247-a792-431d-85cb-9b46982c3373-kube-api-access-nz4mx\") pod \"node-resolver-tzgt2\" (UID: \"c0e3a247-a792-431d-85cb-9b46982c3373\") " pod="openshift-dns/node-resolver-tzgt2" Nov 01 19:58:03.498703 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.496059 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9e564268-fa85-48e6-8d1c-5069cc0f9648-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bltk8\" (UID: \"9e564268-fa85-48e6-8d1c-5069cc0f9648\") " pod="openshift-multus/multus-additional-cni-plugins-bltk8" Nov 01 19:58:03.498703 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.496074 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5a57adad-0c11-4902-be16-b8dfb4e3ee13-etc-sysctl-d\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:58:03.498703 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.496091 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a57adad-0c11-4902-be16-b8dfb4e3ee13-tmp\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:58:03.498703 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.496109 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhv76\" (UniqueName: \"kubernetes.io/projected/62c9da55-b21d-4f8f-a3ec-98e823655351-kube-api-access-fhv76\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:58:03.498703 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.496122 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/91ef232e-99ae-42f2-be58-e17cc42f12ea-etc-selinux\") pod \"aws-ebs-csi-driver-node-fvhjt\" (UID: \"91ef232e-99ae-42f2-be58-e17cc42f12ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvhjt" Nov 01 19:58:03.498703 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.496127 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9e564268-fa85-48e6-8d1c-5069cc0f9648-cni-binary-copy\") pod \"multus-additional-cni-plugins-bltk8\" (UID: \"9e564268-fa85-48e6-8d1c-5069cc0f9648\") " pod="openshift-multus/multus-additional-cni-plugins-bltk8" Nov 01 19:58:03.498703 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.496149 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/62c9da55-b21d-4f8f-a3ec-98e823655351-cni-binary-copy\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:58:03.498703 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.496194 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:58:03.498703 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.496224 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-host-run-multus-certs\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:58:03.498703 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.496335 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5a57adad-0c11-4902-be16-b8dfb4e3ee13-etc-sysctl-d\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:58:03.498703 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.496391 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9e564268-fa85-48e6-8d1c-5069cc0f9648-system-cni-dir\") pod \"multus-additional-cni-plugins-bltk8\" (UID: \"9e564268-fa85-48e6-8d1c-5069cc0f9648\") " pod="openshift-multus/multus-additional-cni-plugins-bltk8" Nov 01 19:58:03.498703 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:03.496407 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 19:58:03.498703 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.496429 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/983a921d-61a5-4668-8e17-1a24b6eaf8b6-host\") pod \"node-ca-77fcl\" (UID: \"983a921d-61a5-4668-8e17-1a24b6eaf8b6\") " pod="openshift-image-registry/node-ca-77fcl" Nov 01 19:58:03.498703 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:03.496454 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2efc5bbf-efd4-4a42-bed8-53f2490bb7bc-original-pull-secret podName:2efc5bbf-efd4-4a42-bed8-53f2490bb7bc nodeName:}" failed. No retries permitted until 2026-11-01 19:58:03.996440431 +0000 UTC m=+2.018166698 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2efc5bbf-efd4-4a42-bed8-53f2490bb7bc-original-pull-secret") pod "global-pull-secret-syncer-k5brd" (UID: "2efc5bbf-efd4-4a42-bed8-53f2490bb7bc") : object "kube-system"/"original-pull-secret" not registered Nov 01 19:58:03.498703 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.496470 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-host-run-netns\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:58:03.498703 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.496496 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-ovn-node-metrics-cert\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:58:03.499467 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.496516 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0e3a247-a792-431d-85cb-9b46982c3373-tmp-dir\") pod \"node-resolver-tzgt2\" (UID: \"c0e3a247-a792-431d-85cb-9b46982c3373\") " pod="openshift-dns/node-resolver-tzgt2" Nov 01 19:58:03.499467 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.496551 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-host-slash\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:58:03.499467 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.496589 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-run-systemd\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:58:03.499467 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.496634 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5a57adad-0c11-4902-be16-b8dfb4e3ee13-etc-sysctl-conf\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:58:03.499467 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.496657 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a57adad-0c11-4902-be16-b8dfb4e3ee13-etc-kubernetes\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:58:03.499467 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.496702 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-multus-socket-dir-parent\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:58:03.499467 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.496724 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-host-var-lib-cni-multus\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:58:03.499467 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.496746 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-hostroot\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:58:03.499467 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.496767 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-run-ovn\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:58:03.499467 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.496816 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9e564268-fa85-48e6-8d1c-5069cc0f9648-cni-binary-copy\") pod \"multus-additional-cni-plugins-bltk8\" (UID: \"9e564268-fa85-48e6-8d1c-5069cc0f9648\") " pod="openshift-multus/multus-additional-cni-plugins-bltk8" Nov 01 19:58:03.499467 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.496827 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-ovnkube-script-lib\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:58:03.499467 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.496852 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98mhk\" (UniqueName: \"kubernetes.io/projected/5a57adad-0c11-4902-be16-b8dfb4e3ee13-kube-api-access-98mhk\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:58:03.499467 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.496864 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-host-run-netns\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:58:03.499467 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.496875 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-host-run-netns\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:58:03.499467 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.496896 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-etc-kubernetes\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:58:03.499467 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.496908 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-run-ovn\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:58:03.499467 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.496922 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5a57adad-0c11-4902-be16-b8dfb4e3ee13-etc-systemd\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:58:03.499467 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.496945 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a57adad-0c11-4902-be16-b8dfb4e3ee13-var-lib-kubelet\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:58:03.500147 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.496972 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f16f57a8-9630-416c-bd95-f5a2c6af2f15-iptables-alerter-script\") pod \"iptables-alerter-dg5c2\" (UID: \"f16f57a8-9630-416c-bd95-f5a2c6af2f15\") " pod="openshift-network-operator/iptables-alerter-dg5c2" Nov 01 19:58:03.500147 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.497209 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0e3a247-a792-431d-85cb-9b46982c3373-tmp-dir\") pod \"node-resolver-tzgt2\" (UID: \"c0e3a247-a792-431d-85cb-9b46982c3373\") " pod="openshift-dns/node-resolver-tzgt2" Nov 01 19:58:03.500147 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.497267 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-host-cni-netd\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:58:03.500147 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.497545 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-multus-socket-dir-parent\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:58:03.500147 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.497599 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-env-overrides\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:58:03.500147 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.497647 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5a57adad-0c11-4902-be16-b8dfb4e3ee13-etc-sysctl-conf\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:58:03.500147 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.497652 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-host-var-lib-cni-multus\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:58:03.500147 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.497684 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a57adad-0c11-4902-be16-b8dfb4e3ee13-etc-kubernetes\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:58:03.500147 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.497734 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-hostroot\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:58:03.500147 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.497764 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-etc-kubernetes\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:58:03.500147 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.497816 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-ovnkube-script-lib\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:58:03.500147 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.497839 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5a57adad-0c11-4902-be16-b8dfb4e3ee13-etc-systemd\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:58:03.500147 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.497919 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a57adad-0c11-4902-be16-b8dfb4e3ee13-var-lib-kubelet\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:58:03.500147 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.497963 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-host-run-netns\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:58:03.500147 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.499180 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8537e78c-5b2c-4966-9558-044ddc39119b-agent-certs\") pod \"konnectivity-agent-rz7h2\" (UID: \"8537e78c-5b2c-4966-9558-044ddc39119b\") " pod="kube-system/konnectivity-agent-rz7h2" Nov 01 19:58:03.500147 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.499471 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-ovn-node-metrics-cert\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:58:03.500147 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.499729 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5a57adad-0c11-4902-be16-b8dfb4e3ee13-etc-tuned\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:58:03.500147 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.500085 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a57adad-0c11-4902-be16-b8dfb4e3ee13-tmp\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:58:03.501208 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:03.501181 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 19:58:03.501208 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:03.501208 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 19:58:03.501366 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:03.501219 2576 projected.go:196] Error preparing data for projected volume kube-api-access-vq4k2 for pod openshift-network-diagnostics/network-check-target-b7mcz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 19:58:03.501366 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:03.501262 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/16e51d04-90d5-484d-9f85-e9d484205220-kube-api-access-vq4k2 podName:16e51d04-90d5-484d-9f85-e9d484205220 nodeName:}" failed. No retries permitted until 2026-11-01 19:58:04.001249839 +0000 UTC m=+2.022976114 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-vq4k2" (UniqueName: "kubernetes.io/projected/16e51d04-90d5-484d-9f85-e9d484205220-kube-api-access-vq4k2") pod "network-check-target-b7mcz" (UID: "16e51d04-90d5-484d-9f85-e9d484205220") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 19:58:03.504186 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.503819 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fntm\" (UniqueName: \"kubernetes.io/projected/983a921d-61a5-4668-8e17-1a24b6eaf8b6-kube-api-access-8fntm\") pod \"node-ca-77fcl\" (UID: \"983a921d-61a5-4668-8e17-1a24b6eaf8b6\") " pod="openshift-image-registry/node-ca-77fcl" Nov 01 19:58:03.504186 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.504150 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhv76\" (UniqueName: \"kubernetes.io/projected/62c9da55-b21d-4f8f-a3ec-98e823655351-kube-api-access-fhv76\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:58:03.504354 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.504250 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dg6b\" (UniqueName: \"kubernetes.io/projected/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-kube-api-access-6dg6b\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:58:03.504466 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.504445 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtnhd\" (UniqueName: \"kubernetes.io/projected/65e6a200-c6f4-4671-9d2e-b0158ff3c5ce-kube-api-access-jtnhd\") pod \"network-metrics-daemon-bvqvw\" (UID: \"65e6a200-c6f4-4671-9d2e-b0158ff3c5ce\") " pod="openshift-multus/network-metrics-daemon-bvqvw" Nov 01 19:58:03.504536 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.504470 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz4mx\" (UniqueName: \"kubernetes.io/projected/c0e3a247-a792-431d-85cb-9b46982c3373-kube-api-access-nz4mx\") pod \"node-resolver-tzgt2\" (UID: \"c0e3a247-a792-431d-85cb-9b46982c3373\") " pod="openshift-dns/node-resolver-tzgt2" Nov 01 19:58:03.504839 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.504819 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z6z7\" (UniqueName: \"kubernetes.io/projected/91ef232e-99ae-42f2-be58-e17cc42f12ea-kube-api-access-8z6z7\") pod \"aws-ebs-csi-driver-node-fvhjt\" (UID: \"91ef232e-99ae-42f2-be58-e17cc42f12ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvhjt" Nov 01 19:58:03.505029 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.505014 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnjs4\" (UniqueName: \"kubernetes.io/projected/9e564268-fa85-48e6-8d1c-5069cc0f9648-kube-api-access-mnjs4\") pod \"multus-additional-cni-plugins-bltk8\" (UID: \"9e564268-fa85-48e6-8d1c-5069cc0f9648\") " pod="openshift-multus/multus-additional-cni-plugins-bltk8" Nov 01 19:58:03.505463 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.505443 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98mhk\" (UniqueName: \"kubernetes.io/projected/5a57adad-0c11-4902-be16-b8dfb4e3ee13-kube-api-access-98mhk\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:58:03.506347 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.506330 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvhjt" Nov 01 19:58:03.519227 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.519200 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-tzgt2" Nov 01 19:58:03.597498 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.597465 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9xlj\" (UniqueName: \"kubernetes.io/projected/f16f57a8-9630-416c-bd95-f5a2c6af2f15-kube-api-access-j9xlj\") pod \"iptables-alerter-dg5c2\" (UID: \"f16f57a8-9630-416c-bd95-f5a2c6af2f15\") " pod="openshift-network-operator/iptables-alerter-dg5c2" Nov 01 19:58:03.597665 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.597564 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f16f57a8-9630-416c-bd95-f5a2c6af2f15-host-slash\") pod \"iptables-alerter-dg5c2\" (UID: \"f16f57a8-9630-416c-bd95-f5a2c6af2f15\") " pod="openshift-network-operator/iptables-alerter-dg5c2" Nov 01 19:58:03.597665 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.597629 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f16f57a8-9630-416c-bd95-f5a2c6af2f15-host-slash\") pod \"iptables-alerter-dg5c2\" (UID: \"f16f57a8-9630-416c-bd95-f5a2c6af2f15\") " pod="openshift-network-operator/iptables-alerter-dg5c2" Nov 01 19:58:03.597665 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.597654 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f16f57a8-9630-416c-bd95-f5a2c6af2f15-iptables-alerter-script\") pod \"iptables-alerter-dg5c2\" (UID: \"f16f57a8-9630-416c-bd95-f5a2c6af2f15\") " pod="openshift-network-operator/iptables-alerter-dg5c2" Nov 01 19:58:03.598526 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.598508 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f16f57a8-9630-416c-bd95-f5a2c6af2f15-iptables-alerter-script\") pod \"iptables-alerter-dg5c2\" (UID: \"f16f57a8-9630-416c-bd95-f5a2c6af2f15\") " pod="openshift-network-operator/iptables-alerter-dg5c2" Nov 01 19:58:03.605412 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.605391 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9xlj\" (UniqueName: \"kubernetes.io/projected/f16f57a8-9630-416c-bd95-f5a2c6af2f15-kube-api-access-j9xlj\") pod \"iptables-alerter-dg5c2\" (UID: \"f16f57a8-9630-416c-bd95-f5a2c6af2f15\") " pod="openshift-network-operator/iptables-alerter-dg5c2" Nov 01 19:58:03.694821 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:03.694786 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd535588bb4025d791b22a311a0d6e9f5.slice/crio-0b5b365de0b6be1ee91aec65b04995e35d9cccda934e4cb6491bcde0f1bcf0b0 WatchSource:0}: Error finding container 0b5b365de0b6be1ee91aec65b04995e35d9cccda934e4cb6491bcde0f1bcf0b0: Status 404 returned error can't find the container with id 0b5b365de0b6be1ee91aec65b04995e35d9cccda934e4cb6491bcde0f1bcf0b0 Nov 01 19:58:03.695095 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:03.695077 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f0999f9286775a451886a78f6ad439c.slice/crio-23796b1cbe6433b0a4bdabc415c684b57ffbafc67eda1d4bbfd8bb577981ba61 WatchSource:0}: Error finding container 23796b1cbe6433b0a4bdabc415c684b57ffbafc67eda1d4bbfd8bb577981ba61: Status 404 returned error can't find the container with id 23796b1cbe6433b0a4bdabc415c684b57ffbafc67eda1d4bbfd8bb577981ba61 Nov 01 19:58:03.700715 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.700700 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 19:58:03.710591 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.710576 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:58:03.715270 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.715254 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:58:03.719106 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:03.719084 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cd0e9b2_272f_46f0_88d3_89f58c5dab49.slice/crio-66ab60379d96b63f8e157420e9e45dd63f147c34d12ebee2405bbee10566620f WatchSource:0}: Error finding container 66ab60379d96b63f8e157420e9e45dd63f147c34d12ebee2405bbee10566620f: Status 404 returned error can't find the container with id 66ab60379d96b63f8e157420e9e45dd63f147c34d12ebee2405bbee10566620f Nov 01 19:58:03.722832 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:03.722814 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a57adad_0c11_4902_be16_b8dfb4e3ee13.slice/crio-64ebebdd16e738a87d048a7c47378c7272ae33b30b07bda575d8a6b20828ff40 WatchSource:0}: Error finding container 64ebebdd16e738a87d048a7c47378c7272ae33b30b07bda575d8a6b20828ff40: Status 404 returned error can't find the container with id 64ebebdd16e738a87d048a7c47378c7272ae33b30b07bda575d8a6b20828ff40 Nov 01 19:58:03.731358 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.731345 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-588fk" Nov 01 19:58:03.739515 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:03.739500 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62c9da55_b21d_4f8f_a3ec_98e823655351.slice/crio-afee1e107bc314024deaf12c05f81a70515de378ec416a05bc0a75c3eb5c478d WatchSource:0}: Error finding container afee1e107bc314024deaf12c05f81a70515de378ec416a05bc0a75c3eb5c478d: Status 404 returned error can't find the container with id afee1e107bc314024deaf12c05f81a70515de378ec416a05bc0a75c3eb5c478d Nov 01 19:58:03.749110 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.749090 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rz7h2" Nov 01 19:58:03.756039 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:03.756020 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8537e78c_5b2c_4966_9558_044ddc39119b.slice/crio-ebf7ce221b34e42276e46bea7df404696c67456b71aa0f2cc734056492424635 WatchSource:0}: Error finding container ebf7ce221b34e42276e46bea7df404696c67456b71aa0f2cc734056492424635: Status 404 returned error can't find the container with id ebf7ce221b34e42276e46bea7df404696c67456b71aa0f2cc734056492424635 Nov 01 19:58:03.762412 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.762398 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bltk8" Nov 01 19:58:03.770037 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:03.770016 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e564268_fa85_48e6_8d1c_5069cc0f9648.slice/crio-9a17db04c7c5522bcea5dd87ba07081512f9792373234ba984f754b9334588ff WatchSource:0}: Error finding container 9a17db04c7c5522bcea5dd87ba07081512f9792373234ba984f754b9334588ff: Status 404 returned error can't find the container with id 9a17db04c7c5522bcea5dd87ba07081512f9792373234ba984f754b9334588ff Nov 01 19:58:03.783786 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.783770 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-77fcl" Nov 01 19:58:03.791482 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:03.791460 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod983a921d_61a5_4668_8e17_1a24b6eaf8b6.slice/crio-4fa5c3abb6e944f8ba628a5f163d42f5e86eb83f848be3bb992e2bca3e6d42c0 WatchSource:0}: Error finding container 4fa5c3abb6e944f8ba628a5f163d42f5e86eb83f848be3bb992e2bca3e6d42c0: Status 404 returned error can't find the container with id 4fa5c3abb6e944f8ba628a5f163d42f5e86eb83f848be3bb992e2bca3e6d42c0 Nov 01 19:58:03.824697 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.824675 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dg5c2" Nov 01 19:58:03.833154 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:03.833133 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf16f57a8_9630_416c_bd95_f5a2c6af2f15.slice/crio-171709489a943ced99b7c4119f166be2a9704fe76d0e017f1d427be14614f522 WatchSource:0}: Error finding container 171709489a943ced99b7c4119f166be2a9704fe76d0e017f1d427be14614f522: Status 404 returned error can't find the container with id 171709489a943ced99b7c4119f166be2a9704fe76d0e017f1d427be14614f522 Nov 01 19:58:03.859160 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:03.859144 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0e3a247_a792_431d_85cb_9b46982c3373.slice/crio-09870c8784bd520f17a437877c0c526f28d0f3380a6e8f5aed986a69fec213db WatchSource:0}: Error finding container 09870c8784bd520f17a437877c0c526f28d0f3380a6e8f5aed986a69fec213db: Status 404 returned error can't find the container with id 09870c8784bd520f17a437877c0c526f28d0f3380a6e8f5aed986a69fec213db Nov 01 19:58:03.920434 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:03.920416 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91ef232e_99ae_42f2_be58_e17cc42f12ea.slice/crio-fb54b0679fdeb76219efab65f1a90e7ad44ae01bad5a0a5df84fc00117c83331 WatchSource:0}: Error finding container fb54b0679fdeb76219efab65f1a90e7ad44ae01bad5a0a5df84fc00117c83331: Status 404 returned error can't find the container with id fb54b0679fdeb76219efab65f1a90e7ad44ae01bad5a0a5df84fc00117c83331 Nov 01 19:58:03.999750 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.999728 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65e6a200-c6f4-4671-9d2e-b0158ff3c5ce-metrics-certs\") pod \"network-metrics-daemon-bvqvw\" (UID: \"65e6a200-c6f4-4671-9d2e-b0158ff3c5ce\") " pod="openshift-multus/network-metrics-daemon-bvqvw" Nov 01 19:58:03.999833 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:03.999765 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2efc5bbf-efd4-4a42-bed8-53f2490bb7bc-original-pull-secret\") pod \"global-pull-secret-syncer-k5brd\" (UID: \"2efc5bbf-efd4-4a42-bed8-53f2490bb7bc\") " pod="kube-system/global-pull-secret-syncer-k5brd" Nov 01 19:58:03.999870 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:03.999851 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 19:58:03.999870 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:03.999854 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 19:58:03.999933 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:03.999889 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2efc5bbf-efd4-4a42-bed8-53f2490bb7bc-original-pull-secret podName:2efc5bbf-efd4-4a42-bed8-53f2490bb7bc nodeName:}" failed. No retries permitted until 2026-11-01 19:58:04.999878771 +0000 UTC m=+3.021605034 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2efc5bbf-efd4-4a42-bed8-53f2490bb7bc-original-pull-secret") pod "global-pull-secret-syncer-k5brd" (UID: "2efc5bbf-efd4-4a42-bed8-53f2490bb7bc") : object "kube-system"/"original-pull-secret" not registered Nov 01 19:58:03.999933 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:03.999901 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65e6a200-c6f4-4671-9d2e-b0158ff3c5ce-metrics-certs podName:65e6a200-c6f4-4671-9d2e-b0158ff3c5ce nodeName:}" failed. No retries permitted until 2026-11-01 19:58:04.999895237 +0000 UTC m=+3.021621499 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/65e6a200-c6f4-4671-9d2e-b0158ff3c5ce-metrics-certs") pod "network-metrics-daemon-bvqvw" (UID: "65e6a200-c6f4-4671-9d2e-b0158ff3c5ce") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 19:58:04.100565 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:04.100544 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq4k2\" (UniqueName: \"kubernetes.io/projected/16e51d04-90d5-484d-9f85-e9d484205220-kube-api-access-vq4k2\") pod \"network-check-target-b7mcz\" (UID: \"16e51d04-90d5-484d-9f85-e9d484205220\") " pod="openshift-network-diagnostics/network-check-target-b7mcz" Nov 01 19:58:04.100703 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:04.100689 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 19:58:04.100755 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:04.100706 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 19:58:04.100755 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:04.100715 2576 projected.go:196] Error preparing data for projected volume kube-api-access-vq4k2 for pod openshift-network-diagnostics/network-check-target-b7mcz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 19:58:04.100821 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:04.100755 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/16e51d04-90d5-484d-9f85-e9d484205220-kube-api-access-vq4k2 podName:16e51d04-90d5-484d-9f85-e9d484205220 nodeName:}" failed. No retries permitted until 2026-11-01 19:58:05.100743316 +0000 UTC m=+3.122469583 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-vq4k2" (UniqueName: "kubernetes.io/projected/16e51d04-90d5-484d-9f85-e9d484205220-kube-api-access-vq4k2") pod "network-check-target-b7mcz" (UID: "16e51d04-90d5-484d-9f85-e9d484205220") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 19:58:04.365566 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:04.365536 2576 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 19:58:04.427592 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:04.427495 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 19:53:03 +0000 UTC" deadline="2028-04-12 09:01:09.12387509 +0000 UTC" Nov 01 19:58:04.427592 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:04.427545 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12661h3m4.696334107s" Nov 01 19:58:04.560252 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:04.560118 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvhjt" event={"ID":"91ef232e-99ae-42f2-be58-e17cc42f12ea","Type":"ContainerStarted","Data":"fb54b0679fdeb76219efab65f1a90e7ad44ae01bad5a0a5df84fc00117c83331"} Nov 01 19:58:04.568505 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:04.568413 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dg5c2" event={"ID":"f16f57a8-9630-416c-bd95-f5a2c6af2f15","Type":"ContainerStarted","Data":"171709489a943ced99b7c4119f166be2a9704fe76d0e017f1d427be14614f522"} Nov 01 19:58:04.573673 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:04.573600 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-77fcl" event={"ID":"983a921d-61a5-4668-8e17-1a24b6eaf8b6","Type":"ContainerStarted","Data":"4fa5c3abb6e944f8ba628a5f163d42f5e86eb83f848be3bb992e2bca3e6d42c0"} Nov 01 19:58:04.578363 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:04.578256 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" event={"ID":"5a57adad-0c11-4902-be16-b8dfb4e3ee13","Type":"ContainerStarted","Data":"64ebebdd16e738a87d048a7c47378c7272ae33b30b07bda575d8a6b20828ff40"} Nov 01 19:58:04.582010 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:04.581988 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" event={"ID":"1cd0e9b2-272f-46f0-88d3-89f58c5dab49","Type":"ContainerStarted","Data":"66ab60379d96b63f8e157420e9e45dd63f147c34d12ebee2405bbee10566620f"} Nov 01 19:58:04.599223 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:04.599193 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-26.us-east-2.compute.internal" event={"ID":"7f0999f9286775a451886a78f6ad439c","Type":"ContainerStarted","Data":"23796b1cbe6433b0a4bdabc415c684b57ffbafc67eda1d4bbfd8bb577981ba61"} Nov 01 19:58:04.606540 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:04.606514 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-26.us-east-2.compute.internal" event={"ID":"d535588bb4025d791b22a311a0d6e9f5","Type":"ContainerStarted","Data":"0b5b365de0b6be1ee91aec65b04995e35d9cccda934e4cb6491bcde0f1bcf0b0"} Nov 01 19:58:04.617333 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:04.617270 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tzgt2" event={"ID":"c0e3a247-a792-431d-85cb-9b46982c3373","Type":"ContainerStarted","Data":"09870c8784bd520f17a437877c0c526f28d0f3380a6e8f5aed986a69fec213db"} Nov 01 19:58:04.620567 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:04.620544 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bltk8" event={"ID":"9e564268-fa85-48e6-8d1c-5069cc0f9648","Type":"ContainerStarted","Data":"9a17db04c7c5522bcea5dd87ba07081512f9792373234ba984f754b9334588ff"} Nov 01 19:58:04.631108 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:04.631089 2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 19:58:04.632860 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:04.632767 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rz7h2" event={"ID":"8537e78c-5b2c-4966-9558-044ddc39119b","Type":"ContainerStarted","Data":"ebf7ce221b34e42276e46bea7df404696c67456b71aa0f2cc734056492424635"} Nov 01 19:58:04.643311 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:04.643274 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-588fk" event={"ID":"62c9da55-b21d-4f8f-a3ec-98e823655351","Type":"ContainerStarted","Data":"afee1e107bc314024deaf12c05f81a70515de378ec416a05bc0a75c3eb5c478d"} Nov 01 19:58:04.654150 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:04.654131 2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 19:58:05.008400 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:05.008360 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65e6a200-c6f4-4671-9d2e-b0158ff3c5ce-metrics-certs\") pod \"network-metrics-daemon-bvqvw\" (UID: \"65e6a200-c6f4-4671-9d2e-b0158ff3c5ce\") " pod="openshift-multus/network-metrics-daemon-bvqvw" Nov 01 19:58:05.008586 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:05.008438 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2efc5bbf-efd4-4a42-bed8-53f2490bb7bc-original-pull-secret\") pod \"global-pull-secret-syncer-k5brd\" (UID: \"2efc5bbf-efd4-4a42-bed8-53f2490bb7bc\") " pod="kube-system/global-pull-secret-syncer-k5brd" Nov 01 19:58:05.008645 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:05.008585 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 19:58:05.008705 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:05.008646 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2efc5bbf-efd4-4a42-bed8-53f2490bb7bc-original-pull-secret podName:2efc5bbf-efd4-4a42-bed8-53f2490bb7bc nodeName:}" failed. No retries permitted until 2026-11-01 19:58:07.008629867 +0000 UTC m=+5.030356133 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2efc5bbf-efd4-4a42-bed8-53f2490bb7bc-original-pull-secret") pod "global-pull-secret-syncer-k5brd" (UID: "2efc5bbf-efd4-4a42-bed8-53f2490bb7bc") : object "kube-system"/"original-pull-secret" not registered Nov 01 19:58:05.009056 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:05.009034 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 19:58:05.009138 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:05.009095 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65e6a200-c6f4-4671-9d2e-b0158ff3c5ce-metrics-certs podName:65e6a200-c6f4-4671-9d2e-b0158ff3c5ce nodeName:}" failed. No retries permitted until 2026-11-01 19:58:07.009079651 +0000 UTC m=+5.030805930 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/65e6a200-c6f4-4671-9d2e-b0158ff3c5ce-metrics-certs") pod "network-metrics-daemon-bvqvw" (UID: "65e6a200-c6f4-4671-9d2e-b0158ff3c5ce") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 19:58:05.108914 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:05.108882 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq4k2\" (UniqueName: \"kubernetes.io/projected/16e51d04-90d5-484d-9f85-e9d484205220-kube-api-access-vq4k2\") pod \"network-check-target-b7mcz\" (UID: \"16e51d04-90d5-484d-9f85-e9d484205220\") " pod="openshift-network-diagnostics/network-check-target-b7mcz" Nov 01 19:58:05.109099 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:05.109037 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 19:58:05.109099 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:05.109056 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 19:58:05.109099 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:05.109072 2576 projected.go:196] Error preparing data for projected volume kube-api-access-vq4k2 for pod openshift-network-diagnostics/network-check-target-b7mcz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 19:58:05.109257 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:05.109129 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/16e51d04-90d5-484d-9f85-e9d484205220-kube-api-access-vq4k2 podName:16e51d04-90d5-484d-9f85-e9d484205220 nodeName:}" failed. No retries permitted until 2026-11-01 19:58:07.109112103 +0000 UTC m=+5.130838371 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-vq4k2" (UniqueName: "kubernetes.io/projected/16e51d04-90d5-484d-9f85-e9d484205220-kube-api-access-vq4k2") pod "network-check-target-b7mcz" (UID: "16e51d04-90d5-484d-9f85-e9d484205220") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 19:58:05.428234 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:05.428146 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 19:53:03 +0000 UTC" deadline="2028-05-07 14:22:08.917569495 +0000 UTC" Nov 01 19:58:05.428234 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:05.428187 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13266h24m3.489387127s" Nov 01 19:58:05.537802 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:05.537772 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k5brd" Nov 01 19:58:05.537982 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:05.537939 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k5brd" podUID="2efc5bbf-efd4-4a42-bed8-53f2490bb7bc" Nov 01 19:58:05.538354 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:05.538324 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b7mcz" Nov 01 19:58:05.538446 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:05.538427 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b7mcz" podUID="16e51d04-90d5-484d-9f85-e9d484205220" Nov 01 19:58:05.538505 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:05.538495 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bvqvw" Nov 01 19:58:05.538592 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:05.538576 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bvqvw" podUID="65e6a200-c6f4-4671-9d2e-b0158ff3c5ce" Nov 01 19:58:07.025539 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:07.025499 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65e6a200-c6f4-4671-9d2e-b0158ff3c5ce-metrics-certs\") pod \"network-metrics-daemon-bvqvw\" (UID: \"65e6a200-c6f4-4671-9d2e-b0158ff3c5ce\") " pod="openshift-multus/network-metrics-daemon-bvqvw" Nov 01 19:58:07.026076 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:07.025579 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2efc5bbf-efd4-4a42-bed8-53f2490bb7bc-original-pull-secret\") pod \"global-pull-secret-syncer-k5brd\" (UID: \"2efc5bbf-efd4-4a42-bed8-53f2490bb7bc\") " pod="kube-system/global-pull-secret-syncer-k5brd" Nov 01 19:58:07.026076 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:07.025716 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 19:58:07.026076 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:07.025770 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2efc5bbf-efd4-4a42-bed8-53f2490bb7bc-original-pull-secret podName:2efc5bbf-efd4-4a42-bed8-53f2490bb7bc nodeName:}" failed. No retries permitted until 2026-11-01 19:58:11.025753016 +0000 UTC m=+9.047479287 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2efc5bbf-efd4-4a42-bed8-53f2490bb7bc-original-pull-secret") pod "global-pull-secret-syncer-k5brd" (UID: "2efc5bbf-efd4-4a42-bed8-53f2490bb7bc") : object "kube-system"/"original-pull-secret" not registered Nov 01 19:58:07.026076 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:07.025835 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 19:58:07.026076 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:07.025883 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65e6a200-c6f4-4671-9d2e-b0158ff3c5ce-metrics-certs podName:65e6a200-c6f4-4671-9d2e-b0158ff3c5ce nodeName:}" failed. No retries permitted until 2026-11-01 19:58:11.025871099 +0000 UTC m=+9.047597378 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/65e6a200-c6f4-4671-9d2e-b0158ff3c5ce-metrics-certs") pod "network-metrics-daemon-bvqvw" (UID: "65e6a200-c6f4-4671-9d2e-b0158ff3c5ce") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 19:58:07.127158 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:07.126541 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq4k2\" (UniqueName: \"kubernetes.io/projected/16e51d04-90d5-484d-9f85-e9d484205220-kube-api-access-vq4k2\") pod \"network-check-target-b7mcz\" (UID: \"16e51d04-90d5-484d-9f85-e9d484205220\") " pod="openshift-network-diagnostics/network-check-target-b7mcz" Nov 01 19:58:07.127158 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:07.126727 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 19:58:07.127158 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:07.126748 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 19:58:07.127158 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:07.126761 2576 projected.go:196] Error preparing data for projected volume kube-api-access-vq4k2 for pod openshift-network-diagnostics/network-check-target-b7mcz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 19:58:07.127158 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:07.126811 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/16e51d04-90d5-484d-9f85-e9d484205220-kube-api-access-vq4k2 podName:16e51d04-90d5-484d-9f85-e9d484205220 nodeName:}" failed. No retries permitted until 2026-11-01 19:58:11.126795131 +0000 UTC m=+9.148521406 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-vq4k2" (UniqueName: "kubernetes.io/projected/16e51d04-90d5-484d-9f85-e9d484205220-kube-api-access-vq4k2") pod "network-check-target-b7mcz" (UID: "16e51d04-90d5-484d-9f85-e9d484205220") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 19:58:07.538105 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:07.538028 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k5brd" Nov 01 19:58:07.538270 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:07.538157 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k5brd" podUID="2efc5bbf-efd4-4a42-bed8-53f2490bb7bc" Nov 01 19:58:07.538776 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:07.538378 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bvqvw" Nov 01 19:58:07.538776 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:07.538482 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bvqvw" podUID="65e6a200-c6f4-4671-9d2e-b0158ff3c5ce" Nov 01 19:58:07.538776 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:07.538545 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b7mcz" Nov 01 19:58:07.538776 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:07.538613 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b7mcz" podUID="16e51d04-90d5-484d-9f85-e9d484205220" Nov 01 19:58:07.664637 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:07.664606 2576 generic.go:356] "Generic (PLEG): container finished" podID="9e564268-fa85-48e6-8d1c-5069cc0f9648" containerID="cbb74d3f8ffe788c98c5d23d01635c377f6e2ce2056c44594e1fc70b46bd9df1" exitCode=0 Nov 01 19:58:07.664863 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:07.664653 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bltk8" event={"ID":"9e564268-fa85-48e6-8d1c-5069cc0f9648","Type":"ContainerDied","Data":"cbb74d3f8ffe788c98c5d23d01635c377f6e2ce2056c44594e1fc70b46bd9df1"} Nov 01 19:58:08.692996 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:08.692757 2576 generic.go:356] "Generic (PLEG): container finished" podID="7f0999f9286775a451886a78f6ad439c" containerID="7b56774f13c2ea90da8bdcd69d2446d18dca5504b87d7df239bde9576c234850" exitCode=0 Nov 01 19:58:08.693527 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:08.693057 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-26.us-east-2.compute.internal" event={"ID":"7f0999f9286775a451886a78f6ad439c","Type":"ContainerDied","Data":"7b56774f13c2ea90da8bdcd69d2446d18dca5504b87d7df239bde9576c234850"} Nov 01 19:58:08.700940 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:08.700826 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-26.us-east-2.compute.internal" event={"ID":"d535588bb4025d791b22a311a0d6e9f5","Type":"ContainerStarted","Data":"bda95036bab0ac9ac16630ae43929f8d6eb3cd2ab1f16a6c01b8fdb8c1beaccc"} Nov 01 19:58:08.703523 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:08.703494 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rz7h2" event={"ID":"8537e78c-5b2c-4966-9558-044ddc39119b","Type":"ContainerStarted","Data":"2553be9b30b80ed0bfe788309b579920537596f3dae58854126692dc83858b3c"} Nov 01 19:58:08.727819 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:08.727764 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-26.us-east-2.compute.internal" podStartSLOduration=6.727748621 podStartE2EDuration="6.727748621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 19:58:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:58:08.727427078 +0000 UTC m=+6.749153356" watchObservedRunningTime="2026-11-01 19:58:08.727748621 +0000 UTC m=+6.749474910" Nov 01 19:58:08.741033 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:08.740839 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-rz7h2" podStartSLOduration=2.356745198 podStartE2EDuration="6.740825006s" totalImagesPullingTime="4.384079808s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 19:58:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:58:08.740212594 +0000 UTC m=+6.761938880" watchObservedRunningTime="2026-11-01 19:58:08.740825006 +0000 UTC m=+6.762551294" Nov 01 19:58:09.539298 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:09.538500 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b7mcz" Nov 01 19:58:09.539298 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:09.538637 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b7mcz" podUID="16e51d04-90d5-484d-9f85-e9d484205220" Nov 01 19:58:09.539298 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:09.538712 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bvqvw" Nov 01 19:58:09.539298 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:09.538795 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bvqvw" podUID="65e6a200-c6f4-4671-9d2e-b0158ff3c5ce" Nov 01 19:58:09.539298 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:09.538846 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k5brd" Nov 01 19:58:09.539298 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:09.538913 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k5brd" podUID="2efc5bbf-efd4-4a42-bed8-53f2490bb7bc" Nov 01 19:58:09.713783 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:09.713551 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-77fcl" event={"ID":"983a921d-61a5-4668-8e17-1a24b6eaf8b6","Type":"ContainerStarted","Data":"8aac99561e122e256b12800dce391bb16bc7517d3e7ae3ea8a292f526d096dee"} Nov 01 19:58:09.718537 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:09.717427 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-26.us-east-2.compute.internal" event={"ID":"7f0999f9286775a451886a78f6ad439c","Type":"ContainerStarted","Data":"d4a6e4405db18f3a12663796380fdbecf6908bb5c7979df9969f18f942132ff6"} Nov 01 19:58:09.744658 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:09.744604 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-77fcl" podStartSLOduration=2.408189682 podStartE2EDuration="7.744585411s" totalImagesPullingTime="5.336395729s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 19:58:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:58:09.729466074 +0000 UTC m=+7.751192360" watchObservedRunningTime="2026-11-01 19:58:09.744585411 +0000 UTC m=+7.766311703" Nov 01 19:58:09.860411 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:09.860332 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-rz7h2" Nov 01 19:58:09.893384 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:09.893352 2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-rz7h2" Nov 01 19:58:09.893944 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:09.893926 2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-rz7h2" Nov 01 19:58:09.911384 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:09.911307 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-26.us-east-2.compute.internal" podStartSLOduration=7.911272389 podStartE2EDuration="7.911272389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 19:58:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:58:09.745776944 +0000 UTC m=+7.767503230" watchObservedRunningTime="2026-11-01 19:58:09.911272389 +0000 UTC m=+7.932998678" Nov 01 19:58:10.724581 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:10.724549 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvhjt" event={"ID":"91ef232e-99ae-42f2-be58-e17cc42f12ea","Type":"ContainerStarted","Data":"fc3f1ed2539c7e50bfc72c3b88b96766ad1d71f3e144fe0e4a6af7811be869ce"} Nov 01 19:58:10.725827 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:10.725803 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-rz7h2" Nov 01 19:58:11.059585 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:11.059550 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2efc5bbf-efd4-4a42-bed8-53f2490bb7bc-original-pull-secret\") pod \"global-pull-secret-syncer-k5brd\" (UID: \"2efc5bbf-efd4-4a42-bed8-53f2490bb7bc\") " pod="kube-system/global-pull-secret-syncer-k5brd" Nov 01 19:58:11.059742 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:11.059636 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65e6a200-c6f4-4671-9d2e-b0158ff3c5ce-metrics-certs\") pod \"network-metrics-daemon-bvqvw\" (UID: \"65e6a200-c6f4-4671-9d2e-b0158ff3c5ce\") " pod="openshift-multus/network-metrics-daemon-bvqvw" Nov 01 19:58:11.059818 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:11.059771 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 19:58:11.059872 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:11.059838 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65e6a200-c6f4-4671-9d2e-b0158ff3c5ce-metrics-certs podName:65e6a200-c6f4-4671-9d2e-b0158ff3c5ce nodeName:}" failed. No retries permitted until 2026-11-01 19:58:19.059820929 +0000 UTC m=+17.081547198 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/65e6a200-c6f4-4671-9d2e-b0158ff3c5ce-metrics-certs") pod "network-metrics-daemon-bvqvw" (UID: "65e6a200-c6f4-4671-9d2e-b0158ff3c5ce") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 19:58:11.060175 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:11.059772 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 19:58:11.060239 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:11.060211 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2efc5bbf-efd4-4a42-bed8-53f2490bb7bc-original-pull-secret podName:2efc5bbf-efd4-4a42-bed8-53f2490bb7bc nodeName:}" failed. No retries permitted until 2026-11-01 19:58:19.0601973 +0000 UTC m=+17.081923577 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2efc5bbf-efd4-4a42-bed8-53f2490bb7bc-original-pull-secret") pod "global-pull-secret-syncer-k5brd" (UID: "2efc5bbf-efd4-4a42-bed8-53f2490bb7bc") : object "kube-system"/"original-pull-secret" not registered Nov 01 19:58:11.161424 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:11.160772 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq4k2\" (UniqueName: \"kubernetes.io/projected/16e51d04-90d5-484d-9f85-e9d484205220-kube-api-access-vq4k2\") pod \"network-check-target-b7mcz\" (UID: \"16e51d04-90d5-484d-9f85-e9d484205220\") " pod="openshift-network-diagnostics/network-check-target-b7mcz" Nov 01 19:58:11.161424 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:11.160943 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 19:58:11.161424 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:11.160966 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 19:58:11.161424 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:11.160980 2576 projected.go:196] Error preparing data for projected volume kube-api-access-vq4k2 for pod openshift-network-diagnostics/network-check-target-b7mcz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 19:58:11.161424 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:11.161034 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/16e51d04-90d5-484d-9f85-e9d484205220-kube-api-access-vq4k2 podName:16e51d04-90d5-484d-9f85-e9d484205220 nodeName:}" failed. No retries permitted until 2026-11-01 19:58:19.161017579 +0000 UTC m=+17.182743858 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-vq4k2" (UniqueName: "kubernetes.io/projected/16e51d04-90d5-484d-9f85-e9d484205220-kube-api-access-vq4k2") pod "network-check-target-b7mcz" (UID: "16e51d04-90d5-484d-9f85-e9d484205220") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 19:58:11.538302 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:11.538260 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k5brd" Nov 01 19:58:11.538474 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:11.538398 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k5brd" podUID="2efc5bbf-efd4-4a42-bed8-53f2490bb7bc" Nov 01 19:58:11.538793 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:11.538767 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b7mcz" Nov 01 19:58:11.538891 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:11.538870 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b7mcz" podUID="16e51d04-90d5-484d-9f85-e9d484205220" Nov 01 19:58:11.538944 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:11.538929 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bvqvw" Nov 01 19:58:11.539044 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:11.539010 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bvqvw" podUID="65e6a200-c6f4-4671-9d2e-b0158ff3c5ce" Nov 01 19:58:11.784799 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:11.784772 2576 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 01 19:58:12.464047 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:12.463950 2576 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T19:58:11.784793932Z","UUID":"c9249fd9-6f1e-43cb-9f33-682bcf69ec39","Handler":null,"Name":"","Endpoint":""} Nov 01 19:58:12.467705 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:12.467607 2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 01 19:58:12.467705 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:12.467636 2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 01 19:58:12.736308 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:12.735691 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-tzgt2" event={"ID":"c0e3a247-a792-431d-85cb-9b46982c3373","Type":"ContainerStarted","Data":"f6e64f65f5767e82bb8c8aed849a6f519b30daec8199c22cb64aafdf868c6cde"} Nov 01 19:58:12.739157 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:12.739128 2576 generic.go:356] "Generic (PLEG): container finished" podID="9e564268-fa85-48e6-8d1c-5069cc0f9648" containerID="d94a19b39b1dcc753a69bcbbcf68a943d04d75dd342964056f010d82b9d30ab7" exitCode=0 Nov 01 19:58:12.739726 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:12.739703 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bltk8" event={"ID":"9e564268-fa85-48e6-8d1c-5069cc0f9648","Type":"ContainerDied","Data":"d94a19b39b1dcc753a69bcbbcf68a943d04d75dd342964056f010d82b9d30ab7"} Nov 01 19:58:12.743608 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:12.743584 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-588fk" event={"ID":"62c9da55-b21d-4f8f-a3ec-98e823655351","Type":"ContainerStarted","Data":"b41bab250fb533d47c15f784c1855891f1fefa9751030022ab3f8d9f1900feff"} Nov 01 19:58:12.746573 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:12.746547 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvhjt" event={"ID":"91ef232e-99ae-42f2-be58-e17cc42f12ea","Type":"ContainerStarted","Data":"0af046f2eb66f6bef2409e42612318c03df09649fcea075e97c543217102a25e"} Nov 01 19:58:12.751280 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:12.751237 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-tzgt2" podStartSLOduration=2.533194098 podStartE2EDuration="10.751221427s" totalImagesPullingTime="8.218027329s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 19:58:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:58:12.750702215 +0000 UTC m=+10.772428501" watchObservedRunningTime="2026-11-01 19:58:12.751221427 +0000 UTC m=+10.772947718" Nov 01 19:58:12.767042 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:12.766994 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-588fk" podStartSLOduration=2.696736612 podStartE2EDuration="10.766978433s" totalImagesPullingTime="8.070241821s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 19:58:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:58:12.76633622 +0000 UTC m=+10.788062500" watchObservedRunningTime="2026-11-01 19:58:12.766978433 +0000 UTC m=+10.788704721" Nov 01 19:58:13.538009 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:13.537982 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b7mcz" Nov 01 19:58:13.538420 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:13.538046 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bvqvw" Nov 01 19:58:13.538420 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:13.538131 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b7mcz" podUID="16e51d04-90d5-484d-9f85-e9d484205220" Nov 01 19:58:13.538420 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:13.538365 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k5brd" Nov 01 19:58:13.538559 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:13.538450 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bvqvw" podUID="65e6a200-c6f4-4671-9d2e-b0158ff3c5ce" Nov 01 19:58:13.538559 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:13.538509 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k5brd" podUID="2efc5bbf-efd4-4a42-bed8-53f2490bb7bc" Nov 01 19:58:13.751177 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:13.751141 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvhjt" event={"ID":"91ef232e-99ae-42f2-be58-e17cc42f12ea","Type":"ContainerStarted","Data":"26dfe8d450b6aa78c96695dae709eea9aac854857391439156be89e19bf48469"} Nov 01 19:58:13.753241 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:13.753215 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dg5c2" event={"ID":"f16f57a8-9630-416c-bd95-f5a2c6af2f15","Type":"ContainerStarted","Data":"da65e04b73cc8261664b8c38ab46492f6ed5301f9c3882c939fb09f07a2e780f"} Nov 01 19:58:13.757536 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:13.756984 2576 generic.go:356] "Generic (PLEG): container finished" podID="9e564268-fa85-48e6-8d1c-5069cc0f9648" containerID="f04c5c5dd003201e88f0c916ccc81b38981e72a118a47cc080c7b7aee0c8bff6" exitCode=0 Nov 01 19:58:13.757536 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:13.757070 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bltk8" event={"ID":"9e564268-fa85-48e6-8d1c-5069cc0f9648","Type":"ContainerDied","Data":"f04c5c5dd003201e88f0c916ccc81b38981e72a118a47cc080c7b7aee0c8bff6"} Nov 01 19:58:13.767219 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:13.767154 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvhjt" podStartSLOduration=3.219346803 podStartE2EDuration="11.767142912s" totalImagesPullingTime="8.547796109s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 19:58:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:58:13.767087053 +0000 UTC m=+11.788813339" watchObservedRunningTime="2026-11-01 19:58:13.767142912 +0000 UTC m=+11.788869198" Nov 01 19:58:13.780496 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:13.780451 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-dg5c2" podStartSLOduration=3.81484413 podStartE2EDuration="11.780438824s" totalImagesPullingTime="7.965594694s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 19:58:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:58:13.780307176 +0000 UTC m=+11.802033459" watchObservedRunningTime="2026-11-01 19:58:13.780438824 +0000 UTC m=+11.802165104" Nov 01 19:58:14.762098 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:14.761798 2576 generic.go:356] "Generic (PLEG): container finished" podID="9e564268-fa85-48e6-8d1c-5069cc0f9648" containerID="f28392decd350e3ab40ca8de6b94d98606e733e9f2029863759e4bb32a6403f8" exitCode=0 Nov 01 19:58:14.762645 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:14.761888 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bltk8" event={"ID":"9e564268-fa85-48e6-8d1c-5069cc0f9648","Type":"ContainerDied","Data":"f28392decd350e3ab40ca8de6b94d98606e733e9f2029863759e4bb32a6403f8"} Nov 01 19:58:14.763803 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:14.763701 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" event={"ID":"5a57adad-0c11-4902-be16-b8dfb4e3ee13","Type":"ContainerStarted","Data":"0686eb9e2fbd79f40e83b110d0dbc8d91cecce0ef3cc4ae74d91c7c64adc4608"} Nov 01 19:58:14.798502 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:14.796476 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" podStartSLOduration=2.477412909 podStartE2EDuration="12.796460978s" totalImagesPullingTime="10.319048069s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 19:58:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:58:14.796450144 +0000 UTC m=+12.818176426" watchObservedRunningTime="2026-11-01 19:58:14.796460978 +0000 UTC m=+12.818187266" Nov 01 19:58:15.537531 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:15.537492 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k5brd" Nov 01 19:58:15.537695 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:15.537597 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bvqvw" Nov 01 19:58:15.537767 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:15.537704 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b7mcz" Nov 01 19:58:15.537767 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:15.537720 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k5brd" podUID="2efc5bbf-efd4-4a42-bed8-53f2490bb7bc" Nov 01 19:58:15.537874 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:15.537809 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b7mcz" podUID="16e51d04-90d5-484d-9f85-e9d484205220" Nov 01 19:58:15.537941 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:15.537920 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bvqvw" podUID="65e6a200-c6f4-4671-9d2e-b0158ff3c5ce" Nov 01 19:58:16.770728 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:16.770694 2576 generic.go:356] "Generic (PLEG): container finished" podID="9e564268-fa85-48e6-8d1c-5069cc0f9648" containerID="80aecb968da642626a5cbe71a2c9b65ba6cdd5d5f71a17143b7ad340682d5243" exitCode=0 Nov 01 19:58:16.771435 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:16.770735 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bltk8" event={"ID":"9e564268-fa85-48e6-8d1c-5069cc0f9648","Type":"ContainerDied","Data":"80aecb968da642626a5cbe71a2c9b65ba6cdd5d5f71a17143b7ad340682d5243"} Nov 01 19:58:17.537607 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:17.537579 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k5brd" Nov 01 19:58:17.537769 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:17.537580 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bvqvw" Nov 01 19:58:17.537769 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:17.537701 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k5brd" podUID="2efc5bbf-efd4-4a42-bed8-53f2490bb7bc" Nov 01 19:58:17.537928 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:17.537776 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bvqvw" podUID="65e6a200-c6f4-4671-9d2e-b0158ff3c5ce" Nov 01 19:58:17.537928 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:17.537588 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b7mcz" Nov 01 19:58:17.537928 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:17.537882 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b7mcz" podUID="16e51d04-90d5-484d-9f85-e9d484205220" Nov 01 19:58:17.778438 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:17.778403 2576 generic.go:356] "Generic (PLEG): container finished" podID="9e564268-fa85-48e6-8d1c-5069cc0f9648" containerID="1d2d55a79a113336883bde47f67a5a39c2b981bc4213b0b217df4ac153356c63" exitCode=0 Nov 01 19:58:17.778936 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:17.778436 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bltk8" event={"ID":"9e564268-fa85-48e6-8d1c-5069cc0f9648","Type":"ContainerDied","Data":"1d2d55a79a113336883bde47f67a5a39c2b981bc4213b0b217df4ac153356c63"} Nov 01 19:58:18.786928 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:18.786690 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bltk8" event={"ID":"9e564268-fa85-48e6-8d1c-5069cc0f9648","Type":"ContainerStarted","Data":"39d9e4a2f78c57d505cf581a2adbab018dc5da9b996035ea7db15c61228c4e4f"} Nov 01 19:58:18.808160 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:18.808115 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-bltk8" podStartSLOduration=6.244011172 podStartE2EDuration="16.808102606s" totalImagesPullingTime="10.564091434s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 19:58:02 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:58:18.80620175 +0000 UTC m=+16.827928076" watchObservedRunningTime="2026-11-01 19:58:18.808102606 +0000 UTC m=+16.829828889" Nov 01 19:58:19.123842 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:19.123777 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2efc5bbf-efd4-4a42-bed8-53f2490bb7bc-original-pull-secret\") pod \"global-pull-secret-syncer-k5brd\" (UID: \"2efc5bbf-efd4-4a42-bed8-53f2490bb7bc\") " pod="kube-system/global-pull-secret-syncer-k5brd" Nov 01 19:58:19.123842 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:19.123831 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65e6a200-c6f4-4671-9d2e-b0158ff3c5ce-metrics-certs\") pod \"network-metrics-daemon-bvqvw\" (UID: \"65e6a200-c6f4-4671-9d2e-b0158ff3c5ce\") " pod="openshift-multus/network-metrics-daemon-bvqvw" Nov 01 19:58:19.124030 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:19.123913 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 19:58:19.124030 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:19.123916 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 19:58:19.124030 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:19.123959 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65e6a200-c6f4-4671-9d2e-b0158ff3c5ce-metrics-certs podName:65e6a200-c6f4-4671-9d2e-b0158ff3c5ce nodeName:}" failed. No retries permitted until 2026-11-01 19:58:35.12394866 +0000 UTC m=+33.145674924 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/65e6a200-c6f4-4671-9d2e-b0158ff3c5ce-metrics-certs") pod "network-metrics-daemon-bvqvw" (UID: "65e6a200-c6f4-4671-9d2e-b0158ff3c5ce") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 19:58:19.124030 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:19.123972 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2efc5bbf-efd4-4a42-bed8-53f2490bb7bc-original-pull-secret podName:2efc5bbf-efd4-4a42-bed8-53f2490bb7bc nodeName:}" failed. No retries permitted until 2026-11-01 19:58:35.123965778 +0000 UTC m=+33.145692042 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2efc5bbf-efd4-4a42-bed8-53f2490bb7bc-original-pull-secret") pod "global-pull-secret-syncer-k5brd" (UID: "2efc5bbf-efd4-4a42-bed8-53f2490bb7bc") : object "kube-system"/"original-pull-secret" not registered Nov 01 19:58:19.224883 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:19.224854 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq4k2\" (UniqueName: \"kubernetes.io/projected/16e51d04-90d5-484d-9f85-e9d484205220-kube-api-access-vq4k2\") pod \"network-check-target-b7mcz\" (UID: \"16e51d04-90d5-484d-9f85-e9d484205220\") " pod="openshift-network-diagnostics/network-check-target-b7mcz" Nov 01 19:58:19.225025 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:19.225002 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 19:58:19.225025 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:19.225020 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 19:58:19.225086 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:19.225029 2576 projected.go:196] Error preparing data for projected volume kube-api-access-vq4k2 for pod openshift-network-diagnostics/network-check-target-b7mcz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 19:58:19.225086 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:19.225078 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/16e51d04-90d5-484d-9f85-e9d484205220-kube-api-access-vq4k2 podName:16e51d04-90d5-484d-9f85-e9d484205220 nodeName:}" failed. No retries permitted until 2026-11-01 19:58:35.225062869 +0000 UTC m=+33.246789135 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-vq4k2" (UniqueName: "kubernetes.io/projected/16e51d04-90d5-484d-9f85-e9d484205220-kube-api-access-vq4k2") pod "network-check-target-b7mcz" (UID: "16e51d04-90d5-484d-9f85-e9d484205220") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 19:58:19.537507 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:19.537480 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bvqvw" Nov 01 19:58:19.537648 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:19.537547 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b7mcz" Nov 01 19:58:19.537648 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:19.537573 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k5brd" Nov 01 19:58:19.537710 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:19.537648 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b7mcz" podUID="16e51d04-90d5-484d-9f85-e9d484205220" Nov 01 19:58:19.537710 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:19.537685 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k5brd" podUID="2efc5bbf-efd4-4a42-bed8-53f2490bb7bc" Nov 01 19:58:19.537763 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:19.537713 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bvqvw" podUID="65e6a200-c6f4-4671-9d2e-b0158ff3c5ce" Nov 01 19:58:20.795417 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:20.795387 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" event={"ID":"1cd0e9b2-272f-46f0-88d3-89f58c5dab49","Type":"ContainerStarted","Data":"fdcbd70a804db5f8319b1a547f488d8fb8f85472ac84ce41c0614cd5a941bd8e"} Nov 01 19:58:20.795815 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:20.795465 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" event={"ID":"1cd0e9b2-272f-46f0-88d3-89f58c5dab49","Type":"ContainerStarted","Data":"bfa45aada87367010b70f9c4ff7d87c8e1210e08f1a40abaa78c5c7c8c011b4b"} Nov 01 19:58:20.795815 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:20.795496 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" event={"ID":"1cd0e9b2-272f-46f0-88d3-89f58c5dab49","Type":"ContainerStarted","Data":"5b40a3f91632025321bf0b5583abfc51b34a5708d23435b2974b371b479a2da9"} Nov 01 19:58:20.795815 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:20.795525 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" event={"ID":"1cd0e9b2-272f-46f0-88d3-89f58c5dab49","Type":"ContainerStarted","Data":"046170f23c912e8a14214df167233829097b88cc4ea5b16f3113d77f7a7a47ee"} Nov 01 19:58:20.795815 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:20.795552 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" event={"ID":"1cd0e9b2-272f-46f0-88d3-89f58c5dab49","Type":"ContainerStarted","Data":"fe1b116fea5e2cf70f7936d879624690a201aba54a179a6a33219215da1fa4d7"} Nov 01 19:58:20.795815 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:20.795577 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" event={"ID":"1cd0e9b2-272f-46f0-88d3-89f58c5dab49","Type":"ContainerStarted","Data":"a9fa7a694905cbafad0421f4f5aad9442ffbebbde2f6fdf9879105a7f57e3206"} Nov 01 19:58:21.537702 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:21.537663 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k5brd" Nov 01 19:58:21.537702 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:21.537689 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b7mcz" Nov 01 19:58:21.537901 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:21.537680 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bvqvw" Nov 01 19:58:21.537901 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:21.537769 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k5brd" podUID="2efc5bbf-efd4-4a42-bed8-53f2490bb7bc" Nov 01 19:58:21.537901 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:21.537888 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bvqvw" podUID="65e6a200-c6f4-4671-9d2e-b0158ff3c5ce" Nov 01 19:58:21.537981 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:21.537965 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b7mcz" podUID="16e51d04-90d5-484d-9f85-e9d484205220" Nov 01 19:58:22.802821 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:22.802782 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" event={"ID":"1cd0e9b2-272f-46f0-88d3-89f58c5dab49","Type":"ContainerStarted","Data":"2991cb13cbcbff4f23537d9741192a75a79050d246fec2d68d0739a7e27a7c86"} Nov 01 19:58:23.538233 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:23.538199 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b7mcz" Nov 01 19:58:23.538380 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:23.538310 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bvqvw" Nov 01 19:58:23.538380 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:23.538323 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b7mcz" podUID="16e51d04-90d5-484d-9f85-e9d484205220" Nov 01 19:58:23.538380 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:23.538333 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k5brd" Nov 01 19:58:23.538492 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:23.538424 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bvqvw" podUID="65e6a200-c6f4-4671-9d2e-b0158ff3c5ce" Nov 01 19:58:23.538521 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:23.538496 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k5brd" podUID="2efc5bbf-efd4-4a42-bed8-53f2490bb7bc" Nov 01 19:58:24.809200 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:24.809164 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" event={"ID":"1cd0e9b2-272f-46f0-88d3-89f58c5dab49","Type":"ContainerStarted","Data":"65b5fcbf2a47db2c9716de194b81712dc7060c7df2eaca59aee9b9715cbc3667"} Nov 01 19:58:24.809857 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:24.809457 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:58:24.823790 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:24.823772 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:58:24.834014 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:24.833981 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" podStartSLOduration=6.693631256 podStartE2EDuration="22.833957699s" totalImagesPullingTime="16.140326443s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 19:58:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:58:24.832469635 +0000 UTC m=+22.854195923" watchObservedRunningTime="2026-11-01 19:58:24.833957699 +0000 UTC m=+22.855683980" Nov 01 19:58:25.537741 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:25.537707 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k5brd" Nov 01 19:58:25.537933 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:25.537787 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bvqvw" Nov 01 19:58:25.537933 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:25.537804 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b7mcz" Nov 01 19:58:25.537933 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:25.537908 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k5brd" podUID="2efc5bbf-efd4-4a42-bed8-53f2490bb7bc" Nov 01 19:58:25.538059 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:25.538043 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bvqvw" podUID="65e6a200-c6f4-4671-9d2e-b0158ff3c5ce" Nov 01 19:58:25.538143 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:25.538123 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b7mcz" podUID="16e51d04-90d5-484d-9f85-e9d484205220" Nov 01 19:58:25.811914 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:25.811847 2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 19:58:25.812467 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:25.812448 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:58:25.827687 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:25.827667 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:58:25.918340 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:25.918313 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-k5brd"] Nov 01 19:58:25.918504 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:25.918421 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k5brd" Nov 01 19:58:25.918564 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:25.918529 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k5brd" podUID="2efc5bbf-efd4-4a42-bed8-53f2490bb7bc" Nov 01 19:58:25.922077 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:25.922054 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bvqvw"] Nov 01 19:58:25.922214 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:25.922149 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bvqvw" Nov 01 19:58:25.922301 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:25.922241 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bvqvw" podUID="65e6a200-c6f4-4671-9d2e-b0158ff3c5ce" Nov 01 19:58:25.923098 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:25.922888 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-b7mcz"] Nov 01 19:58:25.923098 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:25.922981 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b7mcz" Nov 01 19:58:25.923098 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:25.923066 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b7mcz" podUID="16e51d04-90d5-484d-9f85-e9d484205220" Nov 01 19:58:26.814015 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:26.813992 2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 19:58:27.153102 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:27.153036 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:58:27.537714 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:27.537682 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k5brd" Nov 01 19:58:27.537889 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:27.537730 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b7mcz" Nov 01 19:58:27.537889 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:27.537814 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bvqvw" Nov 01 19:58:27.537889 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:27.537815 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k5brd" podUID="2efc5bbf-efd4-4a42-bed8-53f2490bb7bc" Nov 01 19:58:27.537889 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:27.537885 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b7mcz" podUID="16e51d04-90d5-484d-9f85-e9d484205220" Nov 01 19:58:27.538085 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:27.537958 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bvqvw" podUID="65e6a200-c6f4-4671-9d2e-b0158ff3c5ce" Nov 01 19:58:29.537910 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:29.537721 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b7mcz" Nov 01 19:58:29.538351 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:29.537721 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k5brd" Nov 01 19:58:29.538351 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:29.537977 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b7mcz" podUID="16e51d04-90d5-484d-9f85-e9d484205220" Nov 01 19:58:29.538351 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:29.537737 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bvqvw" Nov 01 19:58:29.538351 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:29.538066 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k5brd" podUID="2efc5bbf-efd4-4a42-bed8-53f2490bb7bc" Nov 01 19:58:29.538351 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:29.538169 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bvqvw" podUID="65e6a200-c6f4-4671-9d2e-b0158ff3c5ce" Nov 01 19:58:29.645075 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:29.645050 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tzgt2_c0e3a247-a792-431d-85cb-9b46982c3373/dns-node-resolver/0.log" Nov 01 19:58:30.229481 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:30.229456 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-77fcl_983a921d-61a5-4668-8e17-1a24b6eaf8b6/node-ca/0.log" Nov 01 19:58:31.538026 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:31.537999 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b7mcz" Nov 01 19:58:31.538482 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:31.538125 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bvqvw" Nov 01 19:58:31.538482 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:31.538130 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b7mcz" podUID="16e51d04-90d5-484d-9f85-e9d484205220" Nov 01 19:58:31.538482 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:31.538254 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bvqvw" podUID="65e6a200-c6f4-4671-9d2e-b0158ff3c5ce" Nov 01 19:58:31.538482 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:31.538330 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k5brd" Nov 01 19:58:31.538482 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:31.538414 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k5brd" podUID="2efc5bbf-efd4-4a42-bed8-53f2490bb7bc" Nov 01 19:58:33.537749 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:33.537724 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k5brd" Nov 01 19:58:33.538144 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:33.537810 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k5brd" podUID="2efc5bbf-efd4-4a42-bed8-53f2490bb7bc" Nov 01 19:58:33.538144 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:33.537891 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b7mcz" Nov 01 19:58:33.538144 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:33.537978 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b7mcz" podUID="16e51d04-90d5-484d-9f85-e9d484205220" Nov 01 19:58:33.538144 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:33.538014 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bvqvw" Nov 01 19:58:33.538144 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:33.538086 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bvqvw" podUID="65e6a200-c6f4-4671-9d2e-b0158ff3c5ce" Nov 01 19:58:35.146278 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:35.146244 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65e6a200-c6f4-4671-9d2e-b0158ff3c5ce-metrics-certs\") pod \"network-metrics-daemon-bvqvw\" (UID: \"65e6a200-c6f4-4671-9d2e-b0158ff3c5ce\") " pod="openshift-multus/network-metrics-daemon-bvqvw" Nov 01 19:58:35.146705 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:35.146305 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2efc5bbf-efd4-4a42-bed8-53f2490bb7bc-original-pull-secret\") pod \"global-pull-secret-syncer-k5brd\" (UID: \"2efc5bbf-efd4-4a42-bed8-53f2490bb7bc\") " pod="kube-system/global-pull-secret-syncer-k5brd" Nov 01 19:58:35.146705 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:35.146391 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 01 19:58:35.146705 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:35.146397 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 19:58:35.146705 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:35.146456 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2efc5bbf-efd4-4a42-bed8-53f2490bb7bc-original-pull-secret podName:2efc5bbf-efd4-4a42-bed8-53f2490bb7bc nodeName:}" failed. No retries permitted until 2026-11-01 19:59:07.146441266 +0000 UTC m=+65.168167530 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2efc5bbf-efd4-4a42-bed8-53f2490bb7bc-original-pull-secret") pod "global-pull-secret-syncer-k5brd" (UID: "2efc5bbf-efd4-4a42-bed8-53f2490bb7bc") : object "kube-system"/"original-pull-secret" not registered Nov 01 19:58:35.146705 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:35.146471 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65e6a200-c6f4-4671-9d2e-b0158ff3c5ce-metrics-certs podName:65e6a200-c6f4-4671-9d2e-b0158ff3c5ce nodeName:}" failed. No retries permitted until 2026-11-01 19:59:07.146464211 +0000 UTC m=+65.168190474 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/65e6a200-c6f4-4671-9d2e-b0158ff3c5ce-metrics-certs") pod "network-metrics-daemon-bvqvw" (UID: "65e6a200-c6f4-4671-9d2e-b0158ff3c5ce") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 01 19:58:35.247356 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:35.247325 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq4k2\" (UniqueName: \"kubernetes.io/projected/16e51d04-90d5-484d-9f85-e9d484205220-kube-api-access-vq4k2\") pod \"network-check-target-b7mcz\" (UID: \"16e51d04-90d5-484d-9f85-e9d484205220\") " pod="openshift-network-diagnostics/network-check-target-b7mcz" Nov 01 19:58:35.247494 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:35.247438 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 01 19:58:35.247494 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:35.247453 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 01 19:58:35.247494 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:35.247462 2576 projected.go:196] Error preparing data for projected volume kube-api-access-vq4k2 for pod openshift-network-diagnostics/network-check-target-b7mcz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 19:58:35.247590 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:35.247517 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/16e51d04-90d5-484d-9f85-e9d484205220-kube-api-access-vq4k2 podName:16e51d04-90d5-484d-9f85-e9d484205220 nodeName:}" failed. No retries permitted until 2026-11-01 19:59:07.247506604 +0000 UTC m=+65.269232884 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-vq4k2" (UniqueName: "kubernetes.io/projected/16e51d04-90d5-484d-9f85-e9d484205220-kube-api-access-vq4k2") pod "network-check-target-b7mcz" (UID: "16e51d04-90d5-484d-9f85-e9d484205220") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 01 19:58:35.537717 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:35.537685 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b7mcz" Nov 01 19:58:35.537895 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:35.537760 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k5brd" Nov 01 19:58:35.537895 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:35.537855 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bvqvw" Nov 01 19:58:35.537895 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:35.537860 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b7mcz" podUID="16e51d04-90d5-484d-9f85-e9d484205220" Nov 01 19:58:35.538021 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:35.537954 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bvqvw" podUID="65e6a200-c6f4-4671-9d2e-b0158ff3c5ce" Nov 01 19:58:35.538058 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:35.538021 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k5brd" podUID="2efc5bbf-efd4-4a42-bed8-53f2490bb7bc" Nov 01 19:58:37.537453 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:37.537419 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bvqvw" Nov 01 19:58:37.537453 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:37.537443 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k5brd" Nov 01 19:58:37.537949 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:37.537514 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bvqvw" podUID="65e6a200-c6f4-4671-9d2e-b0158ff3c5ce" Nov 01 19:58:37.537949 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:37.537544 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b7mcz" Nov 01 19:58:37.537949 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:37.537645 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b7mcz" podUID="16e51d04-90d5-484d-9f85-e9d484205220" Nov 01 19:58:37.537949 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:37.537737 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k5brd" podUID="2efc5bbf-efd4-4a42-bed8-53f2490bb7bc" Nov 01 19:58:39.538187 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:39.538158 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k5brd" Nov 01 19:58:39.538609 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:39.538247 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b7mcz" Nov 01 19:58:39.538609 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:39.538272 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bvqvw" Nov 01 19:58:39.538609 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:39.538362 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b7mcz" podUID="16e51d04-90d5-484d-9f85-e9d484205220" Nov 01 19:58:39.538609 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:39.538248 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k5brd" podUID="2efc5bbf-efd4-4a42-bed8-53f2490bb7bc" Nov 01 19:58:39.538609 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:39.538441 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bvqvw" podUID="65e6a200-c6f4-4671-9d2e-b0158ff3c5ce" Nov 01 19:58:41.537463 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:41.537433 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b7mcz" Nov 01 19:58:41.537924 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:41.537543 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bvqvw" Nov 01 19:58:41.537924 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:41.537551 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k5brd" Nov 01 19:58:41.537924 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:41.537554 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b7mcz" podUID="16e51d04-90d5-484d-9f85-e9d484205220" Nov 01 19:58:41.537924 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:41.537608 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k5brd" podUID="2efc5bbf-efd4-4a42-bed8-53f2490bb7bc" Nov 01 19:58:41.537924 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:41.537684 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bvqvw" podUID="65e6a200-c6f4-4671-9d2e-b0158ff3c5ce" Nov 01 19:58:43.538167 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:43.538137 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b7mcz" Nov 01 19:58:43.538565 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:43.538179 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bvqvw" Nov 01 19:58:43.538565 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:43.538238 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k5brd" Nov 01 19:58:43.538565 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:43.538254 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b7mcz" podUID="16e51d04-90d5-484d-9f85-e9d484205220" Nov 01 19:58:43.538709 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:43.538626 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k5brd" podUID="2efc5bbf-efd4-4a42-bed8-53f2490bb7bc" Nov 01 19:58:43.538789 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:43.538767 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bvqvw" podUID="65e6a200-c6f4-4671-9d2e-b0158ff3c5ce" Nov 01 19:58:45.538035 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:45.537992 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k5brd" Nov 01 19:58:45.538035 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:45.538041 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b7mcz" Nov 01 19:58:45.538487 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:45.538087 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bvqvw" Nov 01 19:58:45.538487 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:45.538097 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k5brd" podUID="2efc5bbf-efd4-4a42-bed8-53f2490bb7bc" Nov 01 19:58:45.538487 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:45.538153 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b7mcz" podUID="16e51d04-90d5-484d-9f85-e9d484205220" Nov 01 19:58:45.538487 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:45.538256 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bvqvw" podUID="65e6a200-c6f4-4671-9d2e-b0158ff3c5ce" Nov 01 19:58:47.537896 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:47.537865 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k5brd" Nov 01 19:58:47.537896 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:47.537892 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b7mcz" Nov 01 19:58:47.538308 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:47.537945 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bvqvw" Nov 01 19:58:47.538308 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:47.537958 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k5brd" podUID="2efc5bbf-efd4-4a42-bed8-53f2490bb7bc" Nov 01 19:58:47.538308 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:47.538012 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b7mcz" podUID="16e51d04-90d5-484d-9f85-e9d484205220" Nov 01 19:58:47.538308 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:47.538072 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bvqvw" podUID="65e6a200-c6f4-4671-9d2e-b0158ff3c5ce" Nov 01 19:58:49.537493 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:49.537459 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b7mcz" Nov 01 19:58:49.537493 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:49.537492 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bvqvw" Nov 01 19:58:49.537987 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:49.537496 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k5brd" Nov 01 19:58:49.537987 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:49.537566 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-b7mcz" podUID="16e51d04-90d5-484d-9f85-e9d484205220" Nov 01 19:58:49.537987 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:49.537639 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-k5brd" podUID="2efc5bbf-efd4-4a42-bed8-53f2490bb7bc" Nov 01 19:58:49.537987 ip-10-0-136-26 kubenswrapper[2576]: E1101 19:58:49.537710 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bvqvw" podUID="65e6a200-c6f4-4671-9d2e-b0158ff3c5ce" Nov 01 19:58:50.246812 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:50.246786 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-26.us-east-2.compute.internal" event="NodeReady" Nov 01 19:58:50.246959 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:50.246945 2576 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 01 19:58:50.286683 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:50.286657 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-5lztx"] Nov 01 19:58:50.288524 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:50.288505 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5lztx" Nov 01 19:58:50.290944 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:50.290921 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-tq7l5"] Nov 01 19:58:50.292407 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:50.292385 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tq7l5" Nov 01 19:58:50.294420 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:50.294398 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 01 19:58:50.294604 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:50.294587 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dgkqn\"" Nov 01 19:58:50.294672 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:50.294632 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 01 19:58:50.298509 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:50.298494 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 01 19:58:50.299674 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:50.299656 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5lztx"] Nov 01 19:58:50.302665 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:50.302644 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tq7l5"] Nov 01 19:58:50.308976 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:50.308954 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-r4spt"] Nov 01 19:58:50.310852 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:50.310837 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-r4spt" Nov 01 19:58:50.324520 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:50.324500 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-r4spt"] Nov 01 19:58:50.393456 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:50.393431 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 01 19:58:50.393620 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:50.393461 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-9pms2\"" Nov 01 19:58:50.393620 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:50.393569 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 01 19:58:50.393697 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:50.393678 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 01 19:58:50.393697 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:50.393688 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 01 19:58:50.393754 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:50.393695 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 01 19:58:50.393754 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:50.393696 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 01 19:58:50.393754 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:50.393738 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9xfd7\"" Nov 01 19:58:50.463069 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:50.463048 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58167352-076f-40d7-abee-1c5f48f67566-metrics-tls\") pod \"dns-default-5lztx\" (UID: \"58167352-076f-40d7-abee-1c5f48f67566\") " pod="openshift-dns/dns-default-5lztx" Nov 01 19:58:50.463147 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:50.463070 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58167352-076f-40d7-abee-1c5f48f67566-tmp-dir\") pod \"dns-default-5lztx\" (UID: \"58167352-076f-40d7-abee-1c5f48f67566\") " pod="openshift-dns/dns-default-5lztx" Nov 01 19:58:50.463147 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:50.463087 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/83a0642b-4dfe-4ddb-b506-f0654c7b6d30-data-volume\") pod \"insights-runtime-extractor-r4spt\" (UID: \"83a0642b-4dfe-4ddb-b506-f0654c7b6d30\") " pod="openshift-insights/insights-runtime-extractor-r4spt" Nov 01 19:58:50.463147 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:50.463107 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a225113f-cd49-48a9-b211-595bf5de763a-cert\") pod \"ingress-canary-tq7l5\" (UID: \"a225113f-cd49-48a9-b211-595bf5de763a\") " pod="openshift-ingress-canary/ingress-canary-tq7l5" Nov 01 19:58:50.463236 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:50.463163 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/83a0642b-4dfe-4ddb-b506-f0654c7b6d30-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r4spt\" (UID: \"83a0642b-4dfe-4ddb-b506-f0654c7b6d30\") " pod="openshift-insights/insights-runtime-extractor-r4spt" Nov 01 19:58:50.463271 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:50.463239 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/58167352-076f-40d7-abee-1c5f48f67566-config-volume\") pod \"dns-default-5lztx\" (UID: \"58167352-076f-40d7-abee-1c5f48f67566\") " pod="openshift-dns/dns-default-5lztx" Nov 01 19:58:50.463319 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:50.463302 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/83a0642b-4dfe-4ddb-b506-f0654c7b6d30-crio-socket\") pod \"insights-runtime-extractor-r4spt\" (UID: \"83a0642b-4dfe-4ddb-b506-f0654c7b6d30\") " pod="openshift-insights/insights-runtime-extractor-r4spt" Nov 01 19:58:50.463360 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:50.463348 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chtj4\" (UniqueName: \"kubernetes.io/projected/58167352-076f-40d7-abee-1c5f48f67566-kube-api-access-chtj4\") pod \"dns-default-5lztx\" (UID: \"58167352-076f-40d7-abee-1c5f48f67566\") " pod="openshift-dns/dns-default-5lztx" Nov 01 19:58:50.463389 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:50.463366 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm8bk\" (UniqueName: \"kubernetes.io/projected/83a0642b-4dfe-4ddb-b506-f0654c7b6d30-kube-api-access-bm8bk\") pod \"insights-runtime-extractor-r4spt\" (UID: \"83a0642b-4dfe-4ddb-b506-f0654c7b6d30\") " pod="openshift-insights/insights-runtime-extractor-r4spt" Nov 01 19:58:50.463423 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:50.463393 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/83a0642b-4dfe-4ddb-b506-f0654c7b6d30-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r4spt\" (UID: \"83a0642b-4dfe-4ddb-b506-f0654c7b6d30\") " pod="openshift-insights/insights-runtime-extractor-r4spt" Nov 01 19:58:50.463423 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:50.463417 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs2sl\" (UniqueName: \"kubernetes.io/projected/a225113f-cd49-48a9-b211-595bf5de763a-kube-api-access-fs2sl\") pod \"ingress-canary-tq7l5\" (UID: \"a225113f-cd49-48a9-b211-595bf5de763a\") " pod="openshift-ingress-canary/ingress-canary-tq7l5" Nov 01 19:58:50.564458 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:50.564434 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a225113f-cd49-48a9-b211-595bf5de763a-cert\") pod \"ingress-canary-tq7l5\" (UID: \"a225113f-cd49-48a9-b211-595bf5de763a\") " pod="openshift-ingress-canary/ingress-canary-tq7l5" Nov 01 19:58:50.564458 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:50.564459 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/83a0642b-4dfe-4ddb-b506-f0654c7b6d30-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r4spt\" (UID: \"83a0642b-4dfe-4ddb-b506-f0654c7b6d30\") " pod="openshift-insights/insights-runtime-extractor-r4spt" Nov 01 19:58:50.564883 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:50.564476 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/58167352-076f-40d7-abee-1c5f48f67566-config-volume\") pod \"dns-default-5lztx\" (UID: \"58167352-076f-40d7-abee-1c5f48f67566\") " pod="openshift-dns/dns-default-5lztx" Nov 01 19:58:50.564883 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:50.564493 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/83a0642b-4dfe-4ddb-b506-f0654c7b6d30-crio-socket\") pod \"insights-runtime-extractor-r4spt\" (UID: \"83a0642b-4dfe-4ddb-b506-f0654c7b6d30\") " pod="openshift-insights/insights-runtime-extractor-r4spt" Nov 01 19:58:50.564883 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:50.564509 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chtj4\" (UniqueName: \"kubernetes.io/projected/58167352-076f-40d7-abee-1c5f48f67566-kube-api-access-chtj4\") pod \"dns-default-5lztx\" (UID: \"58167352-076f-40d7-abee-1c5f48f67566\") " pod="openshift-dns/dns-default-5lztx" Nov 01 19:58:50.564883 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:50.564670 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/83a0642b-4dfe-4ddb-b506-f0654c7b6d30-crio-socket\") pod \"insights-runtime-extractor-r4spt\" (UID: \"83a0642b-4dfe-4ddb-b506-f0654c7b6d30\") " pod="openshift-insights/insights-runtime-extractor-r4spt" Nov 01 19:58:50.564883 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:50.564707 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm8bk\" (UniqueName: \"kubernetes.io/projected/83a0642b-4dfe-4ddb-b506-f0654c7b6d30-kube-api-access-bm8bk\") pod \"insights-runtime-extractor-r4spt\" (UID: \"83a0642b-4dfe-4ddb-b506-f0654c7b6d30\") " pod="openshift-insights/insights-runtime-extractor-r4spt" Nov 01 19:58:50.564883 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:50.564739 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/83a0642b-4dfe-4ddb-b506-f0654c7b6d30-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r4spt\" (UID: \"83a0642b-4dfe-4ddb-b506-f0654c7b6d30\") " pod="openshift-insights/insights-runtime-extractor-r4spt" Nov 01 19:58:50.564883 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:50.564761 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs2sl\" (UniqueName: \"kubernetes.io/projected/a225113f-cd49-48a9-b211-595bf5de763a-kube-api-access-fs2sl\") pod \"ingress-canary-tq7l5\" (UID: \"a225113f-cd49-48a9-b211-595bf5de763a\") " pod="openshift-ingress-canary/ingress-canary-tq7l5" Nov 01 19:58:50.564883 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:50.564834 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58167352-076f-40d7-abee-1c5f48f67566-metrics-tls\") pod \"dns-default-5lztx\" (UID: \"58167352-076f-40d7-abee-1c5f48f67566\") " pod="openshift-dns/dns-default-5lztx" Nov 01 19:58:50.564883 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:50.564859 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58167352-076f-40d7-abee-1c5f48f67566-tmp-dir\") pod \"dns-default-5lztx\" (UID: \"58167352-076f-40d7-abee-1c5f48f67566\") " pod="openshift-dns/dns-default-5lztx" Nov 01 19:58:50.564883 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:50.564885 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/83a0642b-4dfe-4ddb-b506-f0654c7b6d30-data-volume\") pod \"insights-runtime-extractor-r4spt\" (UID: \"83a0642b-4dfe-4ddb-b506-f0654c7b6d30\") " pod="openshift-insights/insights-runtime-extractor-r4spt" Nov 01 19:58:50.565338 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:50.565195 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/83a0642b-4dfe-4ddb-b506-f0654c7b6d30-data-volume\") pod \"insights-runtime-extractor-r4spt\" (UID: \"83a0642b-4dfe-4ddb-b506-f0654c7b6d30\") " pod="openshift-insights/insights-runtime-extractor-r4spt" Nov 01 19:58:50.565338 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:50.565239 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/58167352-076f-40d7-abee-1c5f48f67566-config-volume\") pod \"dns-default-5lztx\" (UID: \"58167352-076f-40d7-abee-1c5f48f67566\") " pod="openshift-dns/dns-default-5lztx" Nov 01 19:58:50.565338 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:50.565262 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58167352-076f-40d7-abee-1c5f48f67566-tmp-dir\") pod \"dns-default-5lztx\" (UID: \"58167352-076f-40d7-abee-1c5f48f67566\") " pod="openshift-dns/dns-default-5lztx" Nov 01 19:58:50.565338 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:50.565326 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/83a0642b-4dfe-4ddb-b506-f0654c7b6d30-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r4spt\" (UID: \"83a0642b-4dfe-4ddb-b506-f0654c7b6d30\") " pod="openshift-insights/insights-runtime-extractor-r4spt" Nov 01 19:58:50.567744 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:50.567722 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/83a0642b-4dfe-4ddb-b506-f0654c7b6d30-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r4spt\" (UID: \"83a0642b-4dfe-4ddb-b506-f0654c7b6d30\") " pod="openshift-insights/insights-runtime-extractor-r4spt" Nov 01 19:58:50.567814 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:50.567729 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58167352-076f-40d7-abee-1c5f48f67566-metrics-tls\") pod \"dns-default-5lztx\" (UID: \"58167352-076f-40d7-abee-1c5f48f67566\") " pod="openshift-dns/dns-default-5lztx" Nov 01 19:58:50.567961 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:50.567923 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a225113f-cd49-48a9-b211-595bf5de763a-cert\") pod \"ingress-canary-tq7l5\" (UID: \"a225113f-cd49-48a9-b211-595bf5de763a\") " pod="openshift-ingress-canary/ingress-canary-tq7l5" Nov 01 19:58:50.575350 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:50.572054 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chtj4\" (UniqueName: \"kubernetes.io/projected/58167352-076f-40d7-abee-1c5f48f67566-kube-api-access-chtj4\") pod \"dns-default-5lztx\" (UID: \"58167352-076f-40d7-abee-1c5f48f67566\") " pod="openshift-dns/dns-default-5lztx" Nov 01 19:58:50.575350 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:50.572413 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm8bk\" (UniqueName: \"kubernetes.io/projected/83a0642b-4dfe-4ddb-b506-f0654c7b6d30-kube-api-access-bm8bk\") pod \"insights-runtime-extractor-r4spt\" (UID: \"83a0642b-4dfe-4ddb-b506-f0654c7b6d30\") " pod="openshift-insights/insights-runtime-extractor-r4spt" Nov 01 19:58:50.575350 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:50.572706 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs2sl\" (UniqueName: \"kubernetes.io/projected/a225113f-cd49-48a9-b211-595bf5de763a-kube-api-access-fs2sl\") pod \"ingress-canary-tq7l5\" (UID: \"a225113f-cd49-48a9-b211-595bf5de763a\") " pod="openshift-ingress-canary/ingress-canary-tq7l5" Nov 01 19:58:50.600414 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:50.600392 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5lztx" Nov 01 19:58:50.607715 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:50.607701 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-tq7l5" Nov 01 19:58:50.622943 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:50.621441 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-r4spt" Nov 01 19:58:50.880491 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:50.880405 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-tq7l5"] Nov 01 19:58:50.883915 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:50.883891 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5lztx"] Nov 01 19:58:50.884409 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:50.884385 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda225113f_cd49_48a9_b211_595bf5de763a.slice/crio-3b6395bcdb9ea079988584faf913c7ac86c2a5cb495fcfd707a412590ea159ad WatchSource:0}: Error finding container 3b6395bcdb9ea079988584faf913c7ac86c2a5cb495fcfd707a412590ea159ad: Status 404 returned error can't find the container with id 3b6395bcdb9ea079988584faf913c7ac86c2a5cb495fcfd707a412590ea159ad Nov 01 19:58:50.887365 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:50.887342 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58167352_076f_40d7_abee_1c5f48f67566.slice/crio-e7771648236c3606589fa0b023edcfea1aac1c04baba12589d64b147cd4f6be6 WatchSource:0}: Error finding container e7771648236c3606589fa0b023edcfea1aac1c04baba12589d64b147cd4f6be6: Status 404 returned error can't find the container with id e7771648236c3606589fa0b023edcfea1aac1c04baba12589d64b147cd4f6be6 Nov 01 19:58:50.907542 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:50.907522 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-r4spt"] Nov 01 19:58:50.910298 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:50.910259 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83a0642b_4dfe_4ddb_b506_f0654c7b6d30.slice/crio-56edd4df9c723fdf0a862ebdda0d5030f1d5fab4ebdd28d11ef94b80452b733c WatchSource:0}: Error finding container 56edd4df9c723fdf0a862ebdda0d5030f1d5fab4ebdd28d11ef94b80452b733c: Status 404 returned error can't find the container with id 56edd4df9c723fdf0a862ebdda0d5030f1d5fab4ebdd28d11ef94b80452b733c Nov 01 19:58:51.537689 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:51.537648 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k5brd" Nov 01 19:58:51.537689 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:51.537686 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bvqvw" Nov 01 19:58:51.539317 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:51.538329 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b7mcz" Nov 01 19:58:51.542042 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:51.542019 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 01 19:58:51.544765 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:51.544338 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-xcwxv\"" Nov 01 19:58:51.544765 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:51.544423 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 01 19:58:51.544765 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:51.544604 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 01 19:58:51.544765 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:51.544667 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 01 19:58:51.544765 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:51.544703 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kjp59\"" Nov 01 19:58:51.544765 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:51.544731 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-jt8zj\"" Nov 01 19:58:51.864676 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:51.864514 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r4spt" event={"ID":"83a0642b-4dfe-4ddb-b506-f0654c7b6d30","Type":"ContainerStarted","Data":"743d1d8067169802e59563cf631a5049eec82fd6da81adb1692ab8479b7768f4"} Nov 01 19:58:51.864676 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:51.864581 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r4spt" event={"ID":"83a0642b-4dfe-4ddb-b506-f0654c7b6d30","Type":"ContainerStarted","Data":"56edd4df9c723fdf0a862ebdda0d5030f1d5fab4ebdd28d11ef94b80452b733c"} Nov 01 19:58:51.866489 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:51.866466 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tq7l5" event={"ID":"a225113f-cd49-48a9-b211-595bf5de763a","Type":"ContainerStarted","Data":"3b6395bcdb9ea079988584faf913c7ac86c2a5cb495fcfd707a412590ea159ad"} Nov 01 19:58:51.868591 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:51.868566 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5lztx" event={"ID":"58167352-076f-40d7-abee-1c5f48f67566","Type":"ContainerStarted","Data":"e7771648236c3606589fa0b023edcfea1aac1c04baba12589d64b147cd4f6be6"} Nov 01 19:58:52.062022 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:52.061988 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-6tbtd"] Nov 01 19:58:52.064182 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:52.064155 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-6tbtd" Nov 01 19:58:52.068843 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:52.068820 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 01 19:58:52.069605 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:52.069586 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 01 19:58:52.069703 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:52.069664 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 01 19:58:52.069789 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:52.069775 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-4cjlv\"" Nov 01 19:58:52.069883 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:52.069870 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 01 19:58:52.069956 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:52.069938 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 01 19:58:52.072869 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:52.072845 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-6tbtd"] Nov 01 19:58:52.177411 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:52.177328 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fa82c276-5679-40f4-b7b4-e73eba7de9ef-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-6tbtd\" (UID: \"fa82c276-5679-40f4-b7b4-e73eba7de9ef\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6tbtd" Nov 01 19:58:52.177411 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:52.177384 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa82c276-5679-40f4-b7b4-e73eba7de9ef-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-6tbtd\" (UID: \"fa82c276-5679-40f4-b7b4-e73eba7de9ef\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6tbtd" Nov 01 19:58:52.177622 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:52.177470 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7jcx\" (UniqueName: \"kubernetes.io/projected/fa82c276-5679-40f4-b7b4-e73eba7de9ef-kube-api-access-g7jcx\") pod \"prometheus-operator-9cd89b74c-6tbtd\" (UID: \"fa82c276-5679-40f4-b7b4-e73eba7de9ef\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6tbtd" Nov 01 19:58:52.177622 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:52.177500 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/fa82c276-5679-40f4-b7b4-e73eba7de9ef-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-6tbtd\" (UID: \"fa82c276-5679-40f4-b7b4-e73eba7de9ef\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6tbtd" Nov 01 19:58:52.278777 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:52.278747 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fa82c276-5679-40f4-b7b4-e73eba7de9ef-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-6tbtd\" (UID: \"fa82c276-5679-40f4-b7b4-e73eba7de9ef\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6tbtd" Nov 01 19:58:52.278933 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:52.278787 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa82c276-5679-40f4-b7b4-e73eba7de9ef-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-6tbtd\" (UID: \"fa82c276-5679-40f4-b7b4-e73eba7de9ef\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6tbtd" Nov 01 19:58:52.278933 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:52.278845 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7jcx\" (UniqueName: \"kubernetes.io/projected/fa82c276-5679-40f4-b7b4-e73eba7de9ef-kube-api-access-g7jcx\") pod \"prometheus-operator-9cd89b74c-6tbtd\" (UID: \"fa82c276-5679-40f4-b7b4-e73eba7de9ef\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6tbtd" Nov 01 19:58:52.278933 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:52.278867 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/fa82c276-5679-40f4-b7b4-e73eba7de9ef-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-6tbtd\" (UID: \"fa82c276-5679-40f4-b7b4-e73eba7de9ef\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6tbtd" Nov 01 19:58:52.279652 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:52.279625 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa82c276-5679-40f4-b7b4-e73eba7de9ef-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-6tbtd\" (UID: \"fa82c276-5679-40f4-b7b4-e73eba7de9ef\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6tbtd" Nov 01 19:58:52.281697 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:52.281671 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fa82c276-5679-40f4-b7b4-e73eba7de9ef-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-6tbtd\" (UID: \"fa82c276-5679-40f4-b7b4-e73eba7de9ef\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6tbtd" Nov 01 19:58:52.281855 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:52.281831 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/fa82c276-5679-40f4-b7b4-e73eba7de9ef-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-6tbtd\" (UID: \"fa82c276-5679-40f4-b7b4-e73eba7de9ef\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6tbtd" Nov 01 19:58:52.286528 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:52.286510 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7jcx\" (UniqueName: \"kubernetes.io/projected/fa82c276-5679-40f4-b7b4-e73eba7de9ef-kube-api-access-g7jcx\") pod \"prometheus-operator-9cd89b74c-6tbtd\" (UID: \"fa82c276-5679-40f4-b7b4-e73eba7de9ef\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6tbtd" Nov 01 19:58:52.378831 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:52.378800 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-6tbtd" Nov 01 19:58:52.654996 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:52.654970 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-6tbtd"] Nov 01 19:58:52.661324 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:52.660531 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa82c276_5679_40f4_b7b4_e73eba7de9ef.slice/crio-45dad8cb2d5f132939bc7e5aec60c3e23396ae5e309ff9f31a12cd963cbf8a97 WatchSource:0}: Error finding container 45dad8cb2d5f132939bc7e5aec60c3e23396ae5e309ff9f31a12cd963cbf8a97: Status 404 returned error can't find the container with id 45dad8cb2d5f132939bc7e5aec60c3e23396ae5e309ff9f31a12cd963cbf8a97 Nov 01 19:58:52.873632 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:52.873587 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r4spt" event={"ID":"83a0642b-4dfe-4ddb-b506-f0654c7b6d30","Type":"ContainerStarted","Data":"3365f0b2378b4e928b54e66119f41782a4421b0f10b0107a0851e28e9e23ba05"} Nov 01 19:58:52.875921 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:52.875890 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5lztx" event={"ID":"58167352-076f-40d7-abee-1c5f48f67566","Type":"ContainerStarted","Data":"e0885ac217522f0e2689da605f84a78ed04008975f2934881aafc010dff8b468"} Nov 01 19:58:52.876089 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:52.875945 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5lztx" event={"ID":"58167352-076f-40d7-abee-1c5f48f67566","Type":"ContainerStarted","Data":"543ff381032e30e27a1493c0dabaaa9b120c66bb08c2406f71785ddb25d432da"} Nov 01 19:58:52.876193 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:52.876167 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-5lztx" Nov 01 19:58:52.877585 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:52.877544 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-6tbtd" event={"ID":"fa82c276-5679-40f4-b7b4-e73eba7de9ef","Type":"ContainerStarted","Data":"45dad8cb2d5f132939bc7e5aec60c3e23396ae5e309ff9f31a12cd963cbf8a97"} Nov 01 19:58:52.892219 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:52.892159 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-5lztx" podStartSLOduration=1.201083774 podStartE2EDuration="2.892147719s" totalImagesPullingTime="1.691063945s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 19:58:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:58:52.891101842 +0000 UTC m=+50.912828142" watchObservedRunningTime="2026-11-01 19:58:52.892147719 +0000 UTC m=+50.913874004" Nov 01 19:58:52.946847 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:52.946784 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5lztx_58167352-076f-40d7-abee-1c5f48f67566/dns/0.log" Nov 01 19:58:53.146939 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:53.146885 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5lztx_58167352-076f-40d7-abee-1c5f48f67566/kube-rbac-proxy/0.log" Nov 01 19:58:53.668784 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:53.668747 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-865bfb674b-jlr46"] Nov 01 19:58:53.671505 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:53.671431 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-865bfb674b-jlr46" Nov 01 19:58:53.677727 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:53.677707 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 01 19:58:53.680730 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:53.680709 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 01 19:58:53.684951 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:53.684928 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 01 19:58:53.686155 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:53.685451 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 01 19:58:53.686531 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:53.686339 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-snwvx\"" Nov 01 19:58:53.687239 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:53.687217 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 01 19:58:53.687498 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:53.687482 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-865bfb674b-jlr46"] Nov 01 19:58:53.778033 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:53.778011 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 01 19:58:53.778147 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:53.778040 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 01 19:58:53.778147 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:53.778040 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 01 19:58:53.790392 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:53.790368 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-service-ca\") pod \"console-865bfb674b-jlr46\" (UID: \"87d7d5b7-5e96-4777-afcb-2a26d45b0ba8\") " pod="openshift-console/console-865bfb674b-jlr46" Nov 01 19:58:53.790503 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:53.790408 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-tmp\") pod \"console-865bfb674b-jlr46\" (UID: \"87d7d5b7-5e96-4777-afcb-2a26d45b0ba8\") " pod="openshift-console/console-865bfb674b-jlr46" Nov 01 19:58:53.790503 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:53.790436 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcwgs\" (UniqueName: \"kubernetes.io/projected/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-kube-api-access-pcwgs\") pod \"console-865bfb674b-jlr46\" (UID: \"87d7d5b7-5e96-4777-afcb-2a26d45b0ba8\") " pod="openshift-console/console-865bfb674b-jlr46" Nov 01 19:58:53.790615 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:53.790574 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-console-config\") pod \"console-865bfb674b-jlr46\" (UID: \"87d7d5b7-5e96-4777-afcb-2a26d45b0ba8\") " pod="openshift-console/console-865bfb674b-jlr46" Nov 01 19:58:53.790668 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:53.790622 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-trusted-ca-bundle\") pod \"console-865bfb674b-jlr46\" (UID: \"87d7d5b7-5e96-4777-afcb-2a26d45b0ba8\") " pod="openshift-console/console-865bfb674b-jlr46" Nov 01 19:58:53.790720 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:53.790670 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-console-oauth-config\") pod \"console-865bfb674b-jlr46\" (UID: \"87d7d5b7-5e96-4777-afcb-2a26d45b0ba8\") " pod="openshift-console/console-865bfb674b-jlr46" Nov 01 19:58:53.790720 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:53.790709 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-oauth-serving-cert\") pod \"console-865bfb674b-jlr46\" (UID: \"87d7d5b7-5e96-4777-afcb-2a26d45b0ba8\") " pod="openshift-console/console-865bfb674b-jlr46" Nov 01 19:58:53.790820 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:53.790779 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-console-serving-cert\") pod \"console-865bfb674b-jlr46\" (UID: \"87d7d5b7-5e96-4777-afcb-2a26d45b0ba8\") " pod="openshift-console/console-865bfb674b-jlr46" Nov 01 19:58:53.883762 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:53.883721 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-r4spt" event={"ID":"83a0642b-4dfe-4ddb-b506-f0654c7b6d30","Type":"ContainerStarted","Data":"66e1e42aac71e03bac4812074b68fe2dc7068c55b31d48a414e195590ef90a54"} Nov 01 19:58:53.891784 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:53.891753 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-oauth-serving-cert\") pod \"console-865bfb674b-jlr46\" (UID: \"87d7d5b7-5e96-4777-afcb-2a26d45b0ba8\") " pod="openshift-console/console-865bfb674b-jlr46" Nov 01 19:58:53.891921 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:53.891819 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-console-serving-cert\") pod \"console-865bfb674b-jlr46\" (UID: \"87d7d5b7-5e96-4777-afcb-2a26d45b0ba8\") " pod="openshift-console/console-865bfb674b-jlr46" Nov 01 19:58:53.891921 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:53.891853 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-service-ca\") pod \"console-865bfb674b-jlr46\" (UID: \"87d7d5b7-5e96-4777-afcb-2a26d45b0ba8\") " pod="openshift-console/console-865bfb674b-jlr46" Nov 01 19:58:53.891921 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:53.891879 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-tmp\") pod \"console-865bfb674b-jlr46\" (UID: \"87d7d5b7-5e96-4777-afcb-2a26d45b0ba8\") " pod="openshift-console/console-865bfb674b-jlr46" Nov 01 19:58:53.891921 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:53.891894 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcwgs\" (UniqueName: \"kubernetes.io/projected/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-kube-api-access-pcwgs\") pod \"console-865bfb674b-jlr46\" (UID: \"87d7d5b7-5e96-4777-afcb-2a26d45b0ba8\") " pod="openshift-console/console-865bfb674b-jlr46" Nov 01 19:58:53.891921 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:53.891920 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-console-config\") pod \"console-865bfb674b-jlr46\" (UID: \"87d7d5b7-5e96-4777-afcb-2a26d45b0ba8\") " pod="openshift-console/console-865bfb674b-jlr46" Nov 01 19:58:53.892164 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:53.891953 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-trusted-ca-bundle\") pod \"console-865bfb674b-jlr46\" (UID: \"87d7d5b7-5e96-4777-afcb-2a26d45b0ba8\") " pod="openshift-console/console-865bfb674b-jlr46" Nov 01 19:58:53.892164 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:53.891993 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-console-oauth-config\") pod \"console-865bfb674b-jlr46\" (UID: \"87d7d5b7-5e96-4777-afcb-2a26d45b0ba8\") " pod="openshift-console/console-865bfb674b-jlr46" Nov 01 19:58:53.894082 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:53.892671 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-oauth-serving-cert\") pod \"console-865bfb674b-jlr46\" (UID: \"87d7d5b7-5e96-4777-afcb-2a26d45b0ba8\") " pod="openshift-console/console-865bfb674b-jlr46" Nov 01 19:58:53.894082 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:53.892671 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-tmp\") pod \"console-865bfb674b-jlr46\" (UID: \"87d7d5b7-5e96-4777-afcb-2a26d45b0ba8\") " pod="openshift-console/console-865bfb674b-jlr46" Nov 01 19:58:53.894082 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:53.892951 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-console-config\") pod \"console-865bfb674b-jlr46\" (UID: \"87d7d5b7-5e96-4777-afcb-2a26d45b0ba8\") " pod="openshift-console/console-865bfb674b-jlr46" Nov 01 19:58:53.894082 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:53.893220 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-service-ca\") pod \"console-865bfb674b-jlr46\" (UID: \"87d7d5b7-5e96-4777-afcb-2a26d45b0ba8\") " pod="openshift-console/console-865bfb674b-jlr46" Nov 01 19:58:53.894082 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:53.894035 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-trusted-ca-bundle\") pod \"console-865bfb674b-jlr46\" (UID: \"87d7d5b7-5e96-4777-afcb-2a26d45b0ba8\") " pod="openshift-console/console-865bfb674b-jlr46" Nov 01 19:58:53.895480 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:53.895441 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-console-serving-cert\") pod \"console-865bfb674b-jlr46\" (UID: \"87d7d5b7-5e96-4777-afcb-2a26d45b0ba8\") " pod="openshift-console/console-865bfb674b-jlr46" Nov 01 19:58:53.899937 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:53.899918 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcwgs\" (UniqueName: \"kubernetes.io/projected/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-kube-api-access-pcwgs\") pod \"console-865bfb674b-jlr46\" (UID: \"87d7d5b7-5e96-4777-afcb-2a26d45b0ba8\") " pod="openshift-console/console-865bfb674b-jlr46" Nov 01 19:58:53.900030 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:53.900008 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-console-oauth-config\") pod \"console-865bfb674b-jlr46\" (UID: \"87d7d5b7-5e96-4777-afcb-2a26d45b0ba8\") " pod="openshift-console/console-865bfb674b-jlr46" Nov 01 19:58:53.902863 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:53.902825 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-r4spt" podStartSLOduration=1.235277457 podStartE2EDuration="3.902810029s" totalImagesPullingTime="2.667532572s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 19:58:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:58:53.902588457 +0000 UTC m=+51.924314745" watchObservedRunningTime="2026-11-01 19:58:53.902810029 +0000 UTC m=+51.924536316" Nov 01 19:58:53.986988 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:53.986961 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-865bfb674b-jlr46" Nov 01 19:58:54.273066 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:54.273043 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-865bfb674b-jlr46"] Nov 01 19:58:54.278462 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:54.278429 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87d7d5b7_5e96_4777_afcb_2a26d45b0ba8.slice/crio-e943b81be985021134f7b01fa18e9de2d046185a015ab296c031b2318a819812 WatchSource:0}: Error finding container e943b81be985021134f7b01fa18e9de2d046185a015ab296c031b2318a819812: Status 404 returned error can't find the container with id e943b81be985021134f7b01fa18e9de2d046185a015ab296c031b2318a819812 Nov 01 19:58:54.545727 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:54.545658 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tzgt2_c0e3a247-a792-431d-85cb-9b46982c3373/dns-node-resolver/0.log" Nov 01 19:58:54.889034 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:54.888952 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-865bfb674b-jlr46" event={"ID":"87d7d5b7-5e96-4777-afcb-2a26d45b0ba8","Type":"ContainerStarted","Data":"e943b81be985021134f7b01fa18e9de2d046185a015ab296c031b2318a819812"} Nov 01 19:58:54.891361 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:54.890867 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-6tbtd" event={"ID":"fa82c276-5679-40f4-b7b4-e73eba7de9ef","Type":"ContainerStarted","Data":"f5d39610bd1e3426a1bde74be97d8b2732765a8c3010ee2283c893156203cbf9"} Nov 01 19:58:54.891361 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:54.890926 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-6tbtd" event={"ID":"fa82c276-5679-40f4-b7b4-e73eba7de9ef","Type":"ContainerStarted","Data":"f80bb3c5b08e117f342cb6ff38d38fc4d59ef584afb13d05caa471a9516855de"} Nov 01 19:58:54.892524 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:54.892490 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-tq7l5" event={"ID":"a225113f-cd49-48a9-b211-595bf5de763a","Type":"ContainerStarted","Data":"a37f8d89bcadc7e5ea2f391a6a6aeeb77bf426b1642a14f6e144834f33e434ef"} Nov 01 19:58:54.907293 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:54.907253 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-6tbtd" podStartSLOduration=1.3781945709999999 podStartE2EDuration="2.907242171s" totalImagesPullingTime="1.5290476s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 19:58:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:58:54.906667917 +0000 UTC m=+52.928394204" watchObservedRunningTime="2026-11-01 19:58:54.907242171 +0000 UTC m=+52.928968457" Nov 01 19:58:54.921489 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:54.921449 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-tq7l5" podStartSLOduration=1.5391057240000001 podStartE2EDuration="4.921437722s" totalImagesPullingTime="3.382331998s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 19:58:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:58:54.920652319 +0000 UTC m=+52.942378606" watchObservedRunningTime="2026-11-01 19:58:54.921437722 +0000 UTC m=+52.943164005" Nov 01 19:58:55.345998 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:55.345966 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-77fcl_983a921d-61a5-4668-8e17-1a24b6eaf8b6/node-ca/0.log" Nov 01 19:58:56.454851 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:56.454821 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-z8k8x"] Nov 01 19:58:56.458259 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:56.458236 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-z8k8x" Nov 01 19:58:56.509308 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:56.509257 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 01 19:58:56.509437 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:56.509310 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-88k4l\"" Nov 01 19:58:56.509437 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:56.509348 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 01 19:58:56.512684 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:56.512654 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/961c6886-e297-4202-b10a-df97f3537736-sys\") pod \"node-exporter-z8k8x\" (UID: \"961c6886-e297-4202-b10a-df97f3537736\") " pod="openshift-monitoring/node-exporter-z8k8x" Nov 01 19:58:56.512800 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:56.512705 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/961c6886-e297-4202-b10a-df97f3537736-node-exporter-tls\") pod \"node-exporter-z8k8x\" (UID: \"961c6886-e297-4202-b10a-df97f3537736\") " pod="openshift-monitoring/node-exporter-z8k8x" Nov 01 19:58:56.512874 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:56.512821 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/961c6886-e297-4202-b10a-df97f3537736-node-exporter-textfile\") pod \"node-exporter-z8k8x\" (UID: \"961c6886-e297-4202-b10a-df97f3537736\") " pod="openshift-monitoring/node-exporter-z8k8x" Nov 01 19:58:56.512874 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:56.512867 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/961c6886-e297-4202-b10a-df97f3537736-node-exporter-wtmp\") pod \"node-exporter-z8k8x\" (UID: \"961c6886-e297-4202-b10a-df97f3537736\") " pod="openshift-monitoring/node-exporter-z8k8x" Nov 01 19:58:56.512968 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:56.512890 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/961c6886-e297-4202-b10a-df97f3537736-root\") pod \"node-exporter-z8k8x\" (UID: \"961c6886-e297-4202-b10a-df97f3537736\") " pod="openshift-monitoring/node-exporter-z8k8x" Nov 01 19:58:56.512968 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:56.512952 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/961c6886-e297-4202-b10a-df97f3537736-node-exporter-accelerators-collector-config\") pod \"node-exporter-z8k8x\" (UID: \"961c6886-e297-4202-b10a-df97f3537736\") " pod="openshift-monitoring/node-exporter-z8k8x" Nov 01 19:58:56.513067 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:56.513045 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/961c6886-e297-4202-b10a-df97f3537736-metrics-client-ca\") pod \"node-exporter-z8k8x\" (UID: \"961c6886-e297-4202-b10a-df97f3537736\") " pod="openshift-monitoring/node-exporter-z8k8x" Nov 01 19:58:56.513116 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:56.513072 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/961c6886-e297-4202-b10a-df97f3537736-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-z8k8x\" (UID: \"961c6886-e297-4202-b10a-df97f3537736\") " pod="openshift-monitoring/node-exporter-z8k8x" Nov 01 19:58:56.513116 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:56.513088 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5bdv\" (UniqueName: \"kubernetes.io/projected/961c6886-e297-4202-b10a-df97f3537736-kube-api-access-x5bdv\") pod \"node-exporter-z8k8x\" (UID: \"961c6886-e297-4202-b10a-df97f3537736\") " pod="openshift-monitoring/node-exporter-z8k8x" Nov 01 19:58:56.527269 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:56.527245 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 01 19:58:56.545729 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:56.545708 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-tq7l5_a225113f-cd49-48a9-b211-595bf5de763a/serve-healthcheck-canary/0.log" Nov 01 19:58:56.613747 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:56.613717 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/961c6886-e297-4202-b10a-df97f3537736-sys\") pod \"node-exporter-z8k8x\" (UID: \"961c6886-e297-4202-b10a-df97f3537736\") " pod="openshift-monitoring/node-exporter-z8k8x" Nov 01 19:58:56.613927 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:56.613751 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/961c6886-e297-4202-b10a-df97f3537736-node-exporter-tls\") pod \"node-exporter-z8k8x\" (UID: \"961c6886-e297-4202-b10a-df97f3537736\") " pod="openshift-monitoring/node-exporter-z8k8x" Nov 01 19:58:56.613927 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:56.613788 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/961c6886-e297-4202-b10a-df97f3537736-node-exporter-textfile\") pod \"node-exporter-z8k8x\" (UID: \"961c6886-e297-4202-b10a-df97f3537736\") " pod="openshift-monitoring/node-exporter-z8k8x" Nov 01 19:58:56.613927 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:56.613816 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/961c6886-e297-4202-b10a-df97f3537736-node-exporter-wtmp\") pod \"node-exporter-z8k8x\" (UID: \"961c6886-e297-4202-b10a-df97f3537736\") " pod="openshift-monitoring/node-exporter-z8k8x" Nov 01 19:58:56.613927 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:56.613834 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/961c6886-e297-4202-b10a-df97f3537736-root\") pod \"node-exporter-z8k8x\" (UID: \"961c6886-e297-4202-b10a-df97f3537736\") " pod="openshift-monitoring/node-exporter-z8k8x" Nov 01 19:58:56.613927 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:56.613841 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/961c6886-e297-4202-b10a-df97f3537736-sys\") pod \"node-exporter-z8k8x\" (UID: \"961c6886-e297-4202-b10a-df97f3537736\") " pod="openshift-monitoring/node-exporter-z8k8x" Nov 01 19:58:56.613927 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:56.613911 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/961c6886-e297-4202-b10a-df97f3537736-root\") pod \"node-exporter-z8k8x\" (UID: \"961c6886-e297-4202-b10a-df97f3537736\") " pod="openshift-monitoring/node-exporter-z8k8x" Nov 01 19:58:56.613927 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:56.613913 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/961c6886-e297-4202-b10a-df97f3537736-node-exporter-accelerators-collector-config\") pod \"node-exporter-z8k8x\" (UID: \"961c6886-e297-4202-b10a-df97f3537736\") " pod="openshift-monitoring/node-exporter-z8k8x" Nov 01 19:58:56.614254 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:56.613971 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/961c6886-e297-4202-b10a-df97f3537736-metrics-client-ca\") pod \"node-exporter-z8k8x\" (UID: \"961c6886-e297-4202-b10a-df97f3537736\") " pod="openshift-monitoring/node-exporter-z8k8x" Nov 01 19:58:56.614254 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:56.614007 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/961c6886-e297-4202-b10a-df97f3537736-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-z8k8x\" (UID: \"961c6886-e297-4202-b10a-df97f3537736\") " pod="openshift-monitoring/node-exporter-z8k8x" Nov 01 19:58:56.614254 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:56.614035 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5bdv\" (UniqueName: \"kubernetes.io/projected/961c6886-e297-4202-b10a-df97f3537736-kube-api-access-x5bdv\") pod \"node-exporter-z8k8x\" (UID: \"961c6886-e297-4202-b10a-df97f3537736\") " pod="openshift-monitoring/node-exporter-z8k8x" Nov 01 19:58:56.614254 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:56.614071 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/961c6886-e297-4202-b10a-df97f3537736-node-exporter-wtmp\") pod \"node-exporter-z8k8x\" (UID: \"961c6886-e297-4202-b10a-df97f3537736\") " pod="openshift-monitoring/node-exporter-z8k8x" Nov 01 19:58:56.614254 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:56.614239 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/961c6886-e297-4202-b10a-df97f3537736-node-exporter-textfile\") pod \"node-exporter-z8k8x\" (UID: \"961c6886-e297-4202-b10a-df97f3537736\") " pod="openshift-monitoring/node-exporter-z8k8x" Nov 01 19:58:56.614622 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:56.614599 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/961c6886-e297-4202-b10a-df97f3537736-metrics-client-ca\") pod \"node-exporter-z8k8x\" (UID: \"961c6886-e297-4202-b10a-df97f3537736\") " pod="openshift-monitoring/node-exporter-z8k8x" Nov 01 19:58:56.614746 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:56.614603 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/961c6886-e297-4202-b10a-df97f3537736-node-exporter-accelerators-collector-config\") pod \"node-exporter-z8k8x\" (UID: \"961c6886-e297-4202-b10a-df97f3537736\") " pod="openshift-monitoring/node-exporter-z8k8x" Nov 01 19:58:56.616433 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:56.616414 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/961c6886-e297-4202-b10a-df97f3537736-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-z8k8x\" (UID: \"961c6886-e297-4202-b10a-df97f3537736\") " pod="openshift-monitoring/node-exporter-z8k8x" Nov 01 19:58:56.617836 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:56.617818 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/961c6886-e297-4202-b10a-df97f3537736-node-exporter-tls\") pod \"node-exporter-z8k8x\" (UID: \"961c6886-e297-4202-b10a-df97f3537736\") " pod="openshift-monitoring/node-exporter-z8k8x" Nov 01 19:58:56.621903 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:56.621879 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5bdv\" (UniqueName: \"kubernetes.io/projected/961c6886-e297-4202-b10a-df97f3537736-kube-api-access-x5bdv\") pod \"node-exporter-z8k8x\" (UID: \"961c6886-e297-4202-b10a-df97f3537736\") " pod="openshift-monitoring/node-exporter-z8k8x" Nov 01 19:58:56.775499 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:56.775426 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-z8k8x" Nov 01 19:58:56.790741 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:56.790711 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod961c6886_e297_4202_b10a_df97f3537736.slice/crio-a995ac4070f05fefebeaa4cf8120b1fba5b3082daff99e951f84d285c1583ead WatchSource:0}: Error finding container a995ac4070f05fefebeaa4cf8120b1fba5b3082daff99e951f84d285c1583ead: Status 404 returned error can't find the container with id a995ac4070f05fefebeaa4cf8120b1fba5b3082daff99e951f84d285c1583ead Nov 01 19:58:56.901112 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:56.901068 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z8k8x" event={"ID":"961c6886-e297-4202-b10a-df97f3537736","Type":"ContainerStarted","Data":"a995ac4070f05fefebeaa4cf8120b1fba5b3082daff99e951f84d285c1583ead"} Nov 01 19:58:57.907111 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:57.905975 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-865bfb674b-jlr46" event={"ID":"87d7d5b7-5e96-4777-afcb-2a26d45b0ba8","Type":"ContainerStarted","Data":"e55af4d50013937a201d4ede50c7a2b53a2e9849997d886c823d63c8e96f0130"} Nov 01 19:58:57.907533 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:57.907504 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z8k8x" event={"ID":"961c6886-e297-4202-b10a-df97f3537736","Type":"ContainerStarted","Data":"9c0c31ce88da86d198c89708d2756542617d5bbb78f87a0b31240e9be1ad536f"} Nov 01 19:58:57.923195 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:57.923157 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-865bfb674b-jlr46" podStartSLOduration=2.170452001 podStartE2EDuration="4.923144739s" totalImagesPullingTime="2.752692738s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 19:58:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:58:57.921945948 +0000 UTC m=+55.943672225" watchObservedRunningTime="2026-11-01 19:58:57.923144739 +0000 UTC m=+55.944871016" Nov 01 19:58:58.503011 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:58.502979 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-596fbdd49c-2nmrq"] Nov 01 19:58:58.506329 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:58.506314 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-596fbdd49c-2nmrq" Nov 01 19:58:58.513552 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:58.513529 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 01 19:58:58.513651 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:58.513577 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 01 19:58:58.513777 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:58.513760 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 01 19:58:58.513823 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:58.513774 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-j6j2k\"" Nov 01 19:58:58.513823 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:58.513769 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 01 19:58:58.513823 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:58.513780 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 01 19:58:58.513823 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:58.513815 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-3vfuusupg38va\"" Nov 01 19:58:58.517077 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:58.517057 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-596fbdd49c-2nmrq"] Nov 01 19:58:58.533265 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:58.533241 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/40d57e7e-4896-4e0c-b266-ef5e6ca8beab-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-596fbdd49c-2nmrq\" (UID: \"40d57e7e-4896-4e0c-b266-ef5e6ca8beab\") " pod="openshift-monitoring/thanos-querier-596fbdd49c-2nmrq" Nov 01 19:58:58.533393 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:58.533278 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/40d57e7e-4896-4e0c-b266-ef5e6ca8beab-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-596fbdd49c-2nmrq\" (UID: \"40d57e7e-4896-4e0c-b266-ef5e6ca8beab\") " pod="openshift-monitoring/thanos-querier-596fbdd49c-2nmrq" Nov 01 19:58:58.533393 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:58.533316 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/40d57e7e-4896-4e0c-b266-ef5e6ca8beab-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-596fbdd49c-2nmrq\" (UID: \"40d57e7e-4896-4e0c-b266-ef5e6ca8beab\") " pod="openshift-monitoring/thanos-querier-596fbdd49c-2nmrq" Nov 01 19:58:58.533477 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:58.533440 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/40d57e7e-4896-4e0c-b266-ef5e6ca8beab-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-596fbdd49c-2nmrq\" (UID: \"40d57e7e-4896-4e0c-b266-ef5e6ca8beab\") " pod="openshift-monitoring/thanos-querier-596fbdd49c-2nmrq" Nov 01 19:58:58.533477 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:58.533471 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/40d57e7e-4896-4e0c-b266-ef5e6ca8beab-secret-thanos-querier-tls\") pod \"thanos-querier-596fbdd49c-2nmrq\" (UID: \"40d57e7e-4896-4e0c-b266-ef5e6ca8beab\") " pod="openshift-monitoring/thanos-querier-596fbdd49c-2nmrq" Nov 01 19:58:58.533565 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:58.533489 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/40d57e7e-4896-4e0c-b266-ef5e6ca8beab-metrics-client-ca\") pod \"thanos-querier-596fbdd49c-2nmrq\" (UID: \"40d57e7e-4896-4e0c-b266-ef5e6ca8beab\") " pod="openshift-monitoring/thanos-querier-596fbdd49c-2nmrq" Nov 01 19:58:58.533565 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:58.533508 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/40d57e7e-4896-4e0c-b266-ef5e6ca8beab-secret-grpc-tls\") pod \"thanos-querier-596fbdd49c-2nmrq\" (UID: \"40d57e7e-4896-4e0c-b266-ef5e6ca8beab\") " pod="openshift-monitoring/thanos-querier-596fbdd49c-2nmrq" Nov 01 19:58:58.533669 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:58.533617 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/40d57e7e-4896-4e0c-b266-ef5e6ca8beab-tmp\") pod \"thanos-querier-596fbdd49c-2nmrq\" (UID: \"40d57e7e-4896-4e0c-b266-ef5e6ca8beab\") " pod="openshift-monitoring/thanos-querier-596fbdd49c-2nmrq" Nov 01 19:58:58.533669 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:58.533651 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq6fc\" (UniqueName: \"kubernetes.io/projected/40d57e7e-4896-4e0c-b266-ef5e6ca8beab-kube-api-access-nq6fc\") pod \"thanos-querier-596fbdd49c-2nmrq\" (UID: \"40d57e7e-4896-4e0c-b266-ef5e6ca8beab\") " pod="openshift-monitoring/thanos-querier-596fbdd49c-2nmrq" Nov 01 19:58:58.633946 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:58.633901 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/40d57e7e-4896-4e0c-b266-ef5e6ca8beab-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-596fbdd49c-2nmrq\" (UID: \"40d57e7e-4896-4e0c-b266-ef5e6ca8beab\") " pod="openshift-monitoring/thanos-querier-596fbdd49c-2nmrq" Nov 01 19:58:58.633946 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:58.633951 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/40d57e7e-4896-4e0c-b266-ef5e6ca8beab-secret-thanos-querier-tls\") pod \"thanos-querier-596fbdd49c-2nmrq\" (UID: \"40d57e7e-4896-4e0c-b266-ef5e6ca8beab\") " pod="openshift-monitoring/thanos-querier-596fbdd49c-2nmrq" Nov 01 19:58:58.634154 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:58.633968 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/40d57e7e-4896-4e0c-b266-ef5e6ca8beab-metrics-client-ca\") pod \"thanos-querier-596fbdd49c-2nmrq\" (UID: \"40d57e7e-4896-4e0c-b266-ef5e6ca8beab\") " pod="openshift-monitoring/thanos-querier-596fbdd49c-2nmrq" Nov 01 19:58:58.634154 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:58.633988 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/40d57e7e-4896-4e0c-b266-ef5e6ca8beab-secret-grpc-tls\") pod \"thanos-querier-596fbdd49c-2nmrq\" (UID: \"40d57e7e-4896-4e0c-b266-ef5e6ca8beab\") " pod="openshift-monitoring/thanos-querier-596fbdd49c-2nmrq" Nov 01 19:58:58.634154 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:58.634019 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/40d57e7e-4896-4e0c-b266-ef5e6ca8beab-tmp\") pod \"thanos-querier-596fbdd49c-2nmrq\" (UID: \"40d57e7e-4896-4e0c-b266-ef5e6ca8beab\") " pod="openshift-monitoring/thanos-querier-596fbdd49c-2nmrq" Nov 01 19:58:58.634154 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:58.634035 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq6fc\" (UniqueName: \"kubernetes.io/projected/40d57e7e-4896-4e0c-b266-ef5e6ca8beab-kube-api-access-nq6fc\") pod \"thanos-querier-596fbdd49c-2nmrq\" (UID: \"40d57e7e-4896-4e0c-b266-ef5e6ca8beab\") " pod="openshift-monitoring/thanos-querier-596fbdd49c-2nmrq" Nov 01 19:58:58.634154 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:58.634060 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/40d57e7e-4896-4e0c-b266-ef5e6ca8beab-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-596fbdd49c-2nmrq\" (UID: \"40d57e7e-4896-4e0c-b266-ef5e6ca8beab\") " pod="openshift-monitoring/thanos-querier-596fbdd49c-2nmrq" Nov 01 19:58:58.634154 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:58.634104 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/40d57e7e-4896-4e0c-b266-ef5e6ca8beab-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-596fbdd49c-2nmrq\" (UID: \"40d57e7e-4896-4e0c-b266-ef5e6ca8beab\") " pod="openshift-monitoring/thanos-querier-596fbdd49c-2nmrq" Nov 01 19:58:58.634154 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:58.634132 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/40d57e7e-4896-4e0c-b266-ef5e6ca8beab-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-596fbdd49c-2nmrq\" (UID: \"40d57e7e-4896-4e0c-b266-ef5e6ca8beab\") " pod="openshift-monitoring/thanos-querier-596fbdd49c-2nmrq" Nov 01 19:58:58.634657 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:58.634604 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/40d57e7e-4896-4e0c-b266-ef5e6ca8beab-tmp\") pod \"thanos-querier-596fbdd49c-2nmrq\" (UID: \"40d57e7e-4896-4e0c-b266-ef5e6ca8beab\") " pod="openshift-monitoring/thanos-querier-596fbdd49c-2nmrq" Nov 01 19:58:58.634847 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:58.634798 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/40d57e7e-4896-4e0c-b266-ef5e6ca8beab-metrics-client-ca\") pod \"thanos-querier-596fbdd49c-2nmrq\" (UID: \"40d57e7e-4896-4e0c-b266-ef5e6ca8beab\") " pod="openshift-monitoring/thanos-querier-596fbdd49c-2nmrq" Nov 01 19:58:58.636556 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:58.636531 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/40d57e7e-4896-4e0c-b266-ef5e6ca8beab-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-596fbdd49c-2nmrq\" (UID: \"40d57e7e-4896-4e0c-b266-ef5e6ca8beab\") " pod="openshift-monitoring/thanos-querier-596fbdd49c-2nmrq" Nov 01 19:58:58.636907 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:58.636878 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/40d57e7e-4896-4e0c-b266-ef5e6ca8beab-secret-grpc-tls\") pod \"thanos-querier-596fbdd49c-2nmrq\" (UID: \"40d57e7e-4896-4e0c-b266-ef5e6ca8beab\") " pod="openshift-monitoring/thanos-querier-596fbdd49c-2nmrq" Nov 01 19:58:58.636984 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:58.636957 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/40d57e7e-4896-4e0c-b266-ef5e6ca8beab-secret-thanos-querier-tls\") pod \"thanos-querier-596fbdd49c-2nmrq\" (UID: \"40d57e7e-4896-4e0c-b266-ef5e6ca8beab\") " pod="openshift-monitoring/thanos-querier-596fbdd49c-2nmrq" Nov 01 19:58:58.637030 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:58.636990 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/40d57e7e-4896-4e0c-b266-ef5e6ca8beab-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-596fbdd49c-2nmrq\" (UID: \"40d57e7e-4896-4e0c-b266-ef5e6ca8beab\") " pod="openshift-monitoring/thanos-querier-596fbdd49c-2nmrq" Nov 01 19:58:58.637103 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:58.637087 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/40d57e7e-4896-4e0c-b266-ef5e6ca8beab-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-596fbdd49c-2nmrq\" (UID: \"40d57e7e-4896-4e0c-b266-ef5e6ca8beab\") " pod="openshift-monitoring/thanos-querier-596fbdd49c-2nmrq" Nov 01 19:58:58.637432 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:58.637415 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/40d57e7e-4896-4e0c-b266-ef5e6ca8beab-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-596fbdd49c-2nmrq\" (UID: \"40d57e7e-4896-4e0c-b266-ef5e6ca8beab\") " pod="openshift-monitoring/thanos-querier-596fbdd49c-2nmrq" Nov 01 19:58:58.641817 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:58.641795 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq6fc\" (UniqueName: \"kubernetes.io/projected/40d57e7e-4896-4e0c-b266-ef5e6ca8beab-kube-api-access-nq6fc\") pod \"thanos-querier-596fbdd49c-2nmrq\" (UID: \"40d57e7e-4896-4e0c-b266-ef5e6ca8beab\") " pod="openshift-monitoring/thanos-querier-596fbdd49c-2nmrq" Nov 01 19:58:58.827819 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:58.827755 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-596fbdd49c-2nmrq" Nov 01 19:58:58.827819 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:58.827811 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:58:58.911509 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:58.911478 2576 generic.go:356] "Generic (PLEG): container finished" podID="961c6886-e297-4202-b10a-df97f3537736" containerID="9c0c31ce88da86d198c89708d2756542617d5bbb78f87a0b31240e9be1ad536f" exitCode=0 Nov 01 19:58:58.911889 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:58.911568 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z8k8x" event={"ID":"961c6886-e297-4202-b10a-df97f3537736","Type":"ContainerDied","Data":"9c0c31ce88da86d198c89708d2756542617d5bbb78f87a0b31240e9be1ad536f"} Nov 01 19:58:59.098524 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:59.098366 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-596fbdd49c-2nmrq"] Nov 01 19:58:59.100946 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:58:59.100919 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40d57e7e_4896_4e0c_b266_ef5e6ca8beab.slice/crio-792959e7e29e92310ac6d7f92cafef9566d8b05c6fa9973efacad40daefd3c11 WatchSource:0}: Error finding container 792959e7e29e92310ac6d7f92cafef9566d8b05c6fa9973efacad40daefd3c11: Status 404 returned error can't find the container with id 792959e7e29e92310ac6d7f92cafef9566d8b05c6fa9973efacad40daefd3c11 Nov 01 19:58:59.917514 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:59.917476 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z8k8x" event={"ID":"961c6886-e297-4202-b10a-df97f3537736","Type":"ContainerStarted","Data":"020e6255076cbb3b694425629c8bb60115b77b50018477d4af0d3ace843f0adc"} Nov 01 19:58:59.917975 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:59.917579 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z8k8x" event={"ID":"961c6886-e297-4202-b10a-df97f3537736","Type":"ContainerStarted","Data":"086ad70c0f305cf8c0e65b2af0d68abdbdff05af2b31248b2ecd661434ddc87e"} Nov 01 19:58:59.918897 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:59.918863 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-596fbdd49c-2nmrq" event={"ID":"40d57e7e-4896-4e0c-b266-ef5e6ca8beab","Type":"ContainerStarted","Data":"792959e7e29e92310ac6d7f92cafef9566d8b05c6fa9973efacad40daefd3c11"} Nov 01 19:58:59.938936 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:58:59.938878 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-z8k8x" podStartSLOduration=2.875966102 podStartE2EDuration="3.938861336s" totalImagesPullingTime="998.127177ms" totalInitContainerRuntime="64.768057ms" isStatefulPod=true podCreationTimestamp="2026-11-01 19:58:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:58:59.937688667 +0000 UTC m=+57.959414954" watchObservedRunningTime="2026-11-01 19:58:59.938861336 +0000 UTC m=+57.960587623" Nov 01 19:59:01.927129 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:01.927097 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-596fbdd49c-2nmrq" event={"ID":"40d57e7e-4896-4e0c-b266-ef5e6ca8beab","Type":"ContainerStarted","Data":"b04d6dd85190a16edae5795b7151e8f2f6eb7ca02946cb64f92a5665fbc60c3b"} Nov 01 19:59:01.927533 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:01.927161 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-596fbdd49c-2nmrq" event={"ID":"40d57e7e-4896-4e0c-b266-ef5e6ca8beab","Type":"ContainerStarted","Data":"c078ca4c399448d4bffd72a92643800c8b2c539c173869661b7f0971ca84bf30"} Nov 01 19:59:01.927533 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:01.927185 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-596fbdd49c-2nmrq" event={"ID":"40d57e7e-4896-4e0c-b266-ef5e6ca8beab","Type":"ContainerStarted","Data":"6dc99fcf9459f033b90860adb30c8de402cf1acc227ae61a1f849c65fdbbc40a"} Nov 01 19:59:02.654173 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.652733 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 19:59:02.656698 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.656678 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:02.661612 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.661587 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 01 19:59:02.661823 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.661805 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 01 19:59:02.661910 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.661826 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 01 19:59:02.661910 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.661885 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 01 19:59:02.662007 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.661904 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 01 19:59:02.662007 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.661892 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 01 19:59:02.662007 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.661971 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 01 19:59:02.662007 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.661981 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 01 19:59:02.662007 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.661972 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 01 19:59:02.662007 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.662004 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 01 19:59:02.662190 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.662015 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 01 19:59:02.662190 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.662026 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 01 19:59:02.663355 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.663338 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-dsvm4iuppcnu7\"" Nov 01 19:59:02.663450 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.663362 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 01 19:59:02.663450 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.663384 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 01 19:59:02.663450 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.663416 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-ns62c\"" Nov 01 19:59:02.665739 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.665718 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:02.665998 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.665967 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:02.666183 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.666167 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/80c66ec7-5430-41a1-8641-56db5e4e9d73-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:02.666359 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.666338 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/80c66ec7-5430-41a1-8641-56db5e4e9d73-config-out\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:02.666462 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.666367 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:02.666462 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.666389 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80c66ec7-5430-41a1-8641-56db5e4e9d73-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:02.666462 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.666415 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/80c66ec7-5430-41a1-8641-56db5e4e9d73-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:02.666462 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.666450 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-web-config\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:02.666654 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.666485 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-config\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:02.666654 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.666535 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:02.666654 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.666560 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/80c66ec7-5430-41a1-8641-56db5e4e9d73-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:02.666654 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.666584 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:02.666654 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.666615 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80c66ec7-5430-41a1-8641-56db5e4e9d73-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:02.666654 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.666636 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:02.666904 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.666675 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/80c66ec7-5430-41a1-8641-56db5e4e9d73-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:02.666904 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.666711 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4zhc\" (UniqueName: \"kubernetes.io/projected/80c66ec7-5430-41a1-8641-56db5e4e9d73-kube-api-access-x4zhc\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:02.666904 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.666736 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:02.666904 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.666764 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/80c66ec7-5430-41a1-8641-56db5e4e9d73-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:02.666904 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.666795 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/80c66ec7-5430-41a1-8641-56db5e4e9d73-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:02.666904 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.666819 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80c66ec7-5430-41a1-8641-56db5e4e9d73-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:02.668107 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.668089 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 01 19:59:02.670924 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.670901 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 19:59:02.767464 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.767417 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/80c66ec7-5430-41a1-8641-56db5e4e9d73-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:02.767464 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.767456 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4zhc\" (UniqueName: \"kubernetes.io/projected/80c66ec7-5430-41a1-8641-56db5e4e9d73-kube-api-access-x4zhc\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:02.767464 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.767474 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:02.767761 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.767493 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/80c66ec7-5430-41a1-8641-56db5e4e9d73-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:02.767761 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.767516 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/80c66ec7-5430-41a1-8641-56db5e4e9d73-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:02.767761 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.767542 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80c66ec7-5430-41a1-8641-56db5e4e9d73-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:02.767761 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.767569 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:02.767761 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.767598 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:02.767761 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.767640 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/80c66ec7-5430-41a1-8641-56db5e4e9d73-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:02.767761 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.767665 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/80c66ec7-5430-41a1-8641-56db5e4e9d73-config-out\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:02.767761 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.767686 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:02.767761 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.767706 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80c66ec7-5430-41a1-8641-56db5e4e9d73-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:02.767761 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.767730 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/80c66ec7-5430-41a1-8641-56db5e4e9d73-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:02.767761 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.767764 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-web-config\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:02.768330 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.767803 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-config\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:02.768330 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.767830 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:02.768330 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.767855 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/80c66ec7-5430-41a1-8641-56db5e4e9d73-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:02.768330 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.768153 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/80c66ec7-5430-41a1-8641-56db5e4e9d73-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:02.768330 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.768215 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:02.768330 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.768259 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80c66ec7-5430-41a1-8641-56db5e4e9d73-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:02.768330 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.768308 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:02.768850 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.768829 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/80c66ec7-5430-41a1-8641-56db5e4e9d73-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:02.769929 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.769621 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/80c66ec7-5430-41a1-8641-56db5e4e9d73-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:02.772362 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.772302 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80c66ec7-5430-41a1-8641-56db5e4e9d73-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:02.772362 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.772337 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:02.772611 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.772583 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-web-config\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:02.772952 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.772907 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:02.772952 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.772949 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80c66ec7-5430-41a1-8641-56db5e4e9d73-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:02.773530 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.773509 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/80c66ec7-5430-41a1-8641-56db5e4e9d73-config-out\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:02.773762 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.773739 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/80c66ec7-5430-41a1-8641-56db5e4e9d73-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:02.773833 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.773815 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/80c66ec7-5430-41a1-8641-56db5e4e9d73-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:02.773975 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.773651 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:02.774209 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.774183 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80c66ec7-5430-41a1-8641-56db5e4e9d73-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:02.774335 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.774307 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:02.775418 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.775272 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/80c66ec7-5430-41a1-8641-56db5e4e9d73-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:02.775959 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.775926 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:02.776421 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.776392 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:02.776713 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.776694 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-config\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:02.776861 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.776843 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:02.777487 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.777467 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4zhc\" (UniqueName: \"kubernetes.io/projected/80c66ec7-5430-41a1-8641-56db5e4e9d73-kube-api-access-x4zhc\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:02.885723 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.885692 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5lztx" Nov 01 19:59:02.943494 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.943463 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-596fbdd49c-2nmrq" event={"ID":"40d57e7e-4896-4e0c-b266-ef5e6ca8beab","Type":"ContainerStarted","Data":"7cb2f6e989e5b29403e27b2b7e4242ecc28b604d955b374dc9ef156491d1b6c0"} Nov 01 19:59:02.943952 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.943566 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-596fbdd49c-2nmrq" event={"ID":"40d57e7e-4896-4e0c-b266-ef5e6ca8beab","Type":"ContainerStarted","Data":"bd04fcb87e054c1df6b832744e740c57353e6ade322c88f1d1d183e668b7c297"} Nov 01 19:59:02.943952 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.943611 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-596fbdd49c-2nmrq" event={"ID":"40d57e7e-4896-4e0c-b266-ef5e6ca8beab","Type":"ContainerStarted","Data":"18cf62f0b62c7a0727223ac4ac8bfca2b87a28950bd75ad553d59746b89c8c9e"} Nov 01 19:59:02.943952 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.943660 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-596fbdd49c-2nmrq" Nov 01 19:59:02.964853 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.964798 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-596fbdd49c-2nmrq" podStartSLOduration=1.874456007 podStartE2EDuration="4.964782652s" totalImagesPullingTime="3.090326645s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 19:58:58 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:59:02.963695789 +0000 UTC m=+60.985422086" watchObservedRunningTime="2026-11-01 19:59:02.964782652 +0000 UTC m=+60.986508941" Nov 01 19:59:02.984560 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:02.984537 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:03.271520 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:03.271437 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 19:59:03.274891 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:59:03.274863 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80c66ec7_5430_41a1_8641_56db5e4e9d73.slice/crio-21eeb2fddeba4c6a0109132af0db2fdc6635c8a149fcaa1750e43d852575ed2f WatchSource:0}: Error finding container 21eeb2fddeba4c6a0109132af0db2fdc6635c8a149fcaa1750e43d852575ed2f: Status 404 returned error can't find the container with id 21eeb2fddeba4c6a0109132af0db2fdc6635c8a149fcaa1750e43d852575ed2f Nov 01 19:59:03.875542 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:03.875506 2576 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-865bfb674b-jlr46"] Nov 01 19:59:03.948109 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:03.948073 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"80c66ec7-5430-41a1-8641-56db5e4e9d73","Type":"ContainerStarted","Data":"21eeb2fddeba4c6a0109132af0db2fdc6635c8a149fcaa1750e43d852575ed2f"} Nov 01 19:59:03.987897 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:03.987871 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-865bfb674b-jlr46" Nov 01 19:59:04.952185 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:04.952147 2576 generic.go:356] "Generic (PLEG): container finished" podID="80c66ec7-5430-41a1-8641-56db5e4e9d73" containerID="86c3f1c47b655a4accd6f8c872ff24bcfd277f8e89dd328467251033cab7562f" exitCode=0 Nov 01 19:59:04.952658 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:04.952191 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"80c66ec7-5430-41a1-8641-56db5e4e9d73","Type":"ContainerDied","Data":"86c3f1c47b655a4accd6f8c872ff24bcfd277f8e89dd328467251033cab7562f"} Nov 01 19:59:07.211980 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:07.211948 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2efc5bbf-efd4-4a42-bed8-53f2490bb7bc-original-pull-secret\") pod \"global-pull-secret-syncer-k5brd\" (UID: \"2efc5bbf-efd4-4a42-bed8-53f2490bb7bc\") " pod="kube-system/global-pull-secret-syncer-k5brd" Nov 01 19:59:07.212414 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:07.211997 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65e6a200-c6f4-4671-9d2e-b0158ff3c5ce-metrics-certs\") pod \"network-metrics-daemon-bvqvw\" (UID: \"65e6a200-c6f4-4671-9d2e-b0158ff3c5ce\") " pod="openshift-multus/network-metrics-daemon-bvqvw" Nov 01 19:59:07.215647 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:07.215628 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 01 19:59:07.216139 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:07.216120 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 01 19:59:07.230201 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:07.225416 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2efc5bbf-efd4-4a42-bed8-53f2490bb7bc-original-pull-secret\") pod \"global-pull-secret-syncer-k5brd\" (UID: \"2efc5bbf-efd4-4a42-bed8-53f2490bb7bc\") " pod="kube-system/global-pull-secret-syncer-k5brd" Nov 01 19:59:07.230201 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:07.225416 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65e6a200-c6f4-4671-9d2e-b0158ff3c5ce-metrics-certs\") pod \"network-metrics-daemon-bvqvw\" (UID: \"65e6a200-c6f4-4671-9d2e-b0158ff3c5ce\") " pod="openshift-multus/network-metrics-daemon-bvqvw" Nov 01 19:59:07.312946 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:07.312912 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq4k2\" (UniqueName: \"kubernetes.io/projected/16e51d04-90d5-484d-9f85-e9d484205220-kube-api-access-vq4k2\") pod \"network-check-target-b7mcz\" (UID: \"16e51d04-90d5-484d-9f85-e9d484205220\") " pod="openshift-network-diagnostics/network-check-target-b7mcz" Nov 01 19:59:07.315754 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:07.315735 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 01 19:59:07.326354 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:07.326336 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 01 19:59:07.336669 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:07.336644 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq4k2\" (UniqueName: \"kubernetes.io/projected/16e51d04-90d5-484d-9f85-e9d484205220-kube-api-access-vq4k2\") pod \"network-check-target-b7mcz\" (UID: \"16e51d04-90d5-484d-9f85-e9d484205220\") " pod="openshift-network-diagnostics/network-check-target-b7mcz" Nov 01 19:59:07.455034 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:07.455004 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-jt8zj\"" Nov 01 19:59:07.463212 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:07.463135 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-k5brd" Nov 01 19:59:07.463485 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:07.463466 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kjp59\"" Nov 01 19:59:07.468262 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:07.468247 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-xcwxv\"" Nov 01 19:59:07.471403 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:07.471377 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bvqvw" Nov 01 19:59:07.476440 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:07.476420 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-b7mcz" Nov 01 19:59:07.741112 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:07.741030 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-k5brd"] Nov 01 19:59:07.747832 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:59:07.747798 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2efc5bbf_efd4_4a42_bed8_53f2490bb7bc.slice/crio-849c0f8d3b237ee06cab1e3c239405786fcf922f40061978e92bdd14c3ce4172 WatchSource:0}: Error finding container 849c0f8d3b237ee06cab1e3c239405786fcf922f40061978e92bdd14c3ce4172: Status 404 returned error can't find the container with id 849c0f8d3b237ee06cab1e3c239405786fcf922f40061978e92bdd14c3ce4172 Nov 01 19:59:07.774627 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:07.774507 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bvqvw"] Nov 01 19:59:07.777256 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:07.777227 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-b7mcz"] Nov 01 19:59:07.777519 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:59:07.777496 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65e6a200_c6f4_4671_9d2e_b0158ff3c5ce.slice/crio-1eb923680684f91bc5bb79f36fc319787360453b2d2fdf254d2ed40e1b919e45 WatchSource:0}: Error finding container 1eb923680684f91bc5bb79f36fc319787360453b2d2fdf254d2ed40e1b919e45: Status 404 returned error can't find the container with id 1eb923680684f91bc5bb79f36fc319787360453b2d2fdf254d2ed40e1b919e45 Nov 01 19:59:07.963627 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:07.963590 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-k5brd" event={"ID":"2efc5bbf-efd4-4a42-bed8-53f2490bb7bc","Type":"ContainerStarted","Data":"849c0f8d3b237ee06cab1e3c239405786fcf922f40061978e92bdd14c3ce4172"} Nov 01 19:59:07.964914 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:07.964881 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bvqvw" event={"ID":"65e6a200-c6f4-4671-9d2e-b0158ff3c5ce","Type":"ContainerStarted","Data":"1eb923680684f91bc5bb79f36fc319787360453b2d2fdf254d2ed40e1b919e45"} Nov 01 19:59:07.965952 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:07.965930 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-b7mcz" event={"ID":"16e51d04-90d5-484d-9f85-e9d484205220","Type":"ContainerStarted","Data":"9553a008d6925c67e4a2b603ebe7471784b7851172e5102a19c033372f1fad59"} Nov 01 19:59:08.709863 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:08.709825 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7698dd44-8ddgn"] Nov 01 19:59:08.713244 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:08.713216 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7698dd44-8ddgn" Nov 01 19:59:08.722677 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:08.722653 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7698dd44-8ddgn"] Nov 01 19:59:08.827382 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:08.827340 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cq6h\" (UniqueName: \"kubernetes.io/projected/60e3a171-27e7-4ae6-ad6c-89984ecf5634-kube-api-access-4cq6h\") pod \"console-7698dd44-8ddgn\" (UID: \"60e3a171-27e7-4ae6-ad6c-89984ecf5634\") " pod="openshift-console/console-7698dd44-8ddgn" Nov 01 19:59:08.827563 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:08.827400 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/60e3a171-27e7-4ae6-ad6c-89984ecf5634-console-oauth-config\") pod \"console-7698dd44-8ddgn\" (UID: \"60e3a171-27e7-4ae6-ad6c-89984ecf5634\") " pod="openshift-console/console-7698dd44-8ddgn" Nov 01 19:59:08.827563 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:08.827425 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/60e3a171-27e7-4ae6-ad6c-89984ecf5634-console-config\") pod \"console-7698dd44-8ddgn\" (UID: \"60e3a171-27e7-4ae6-ad6c-89984ecf5634\") " pod="openshift-console/console-7698dd44-8ddgn" Nov 01 19:59:08.827682 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:08.827568 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60e3a171-27e7-4ae6-ad6c-89984ecf5634-trusted-ca-bundle\") pod \"console-7698dd44-8ddgn\" (UID: \"60e3a171-27e7-4ae6-ad6c-89984ecf5634\") " pod="openshift-console/console-7698dd44-8ddgn" Nov 01 19:59:08.827682 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:08.827636 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/60e3a171-27e7-4ae6-ad6c-89984ecf5634-oauth-serving-cert\") pod \"console-7698dd44-8ddgn\" (UID: \"60e3a171-27e7-4ae6-ad6c-89984ecf5634\") " pod="openshift-console/console-7698dd44-8ddgn" Nov 01 19:59:08.827789 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:08.827683 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/60e3a171-27e7-4ae6-ad6c-89984ecf5634-service-ca\") pod \"console-7698dd44-8ddgn\" (UID: \"60e3a171-27e7-4ae6-ad6c-89984ecf5634\") " pod="openshift-console/console-7698dd44-8ddgn" Nov 01 19:59:08.827789 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:08.827716 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/60e3a171-27e7-4ae6-ad6c-89984ecf5634-tmp\") pod \"console-7698dd44-8ddgn\" (UID: \"60e3a171-27e7-4ae6-ad6c-89984ecf5634\") " pod="openshift-console/console-7698dd44-8ddgn" Nov 01 19:59:08.827789 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:08.827775 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/60e3a171-27e7-4ae6-ad6c-89984ecf5634-console-serving-cert\") pod \"console-7698dd44-8ddgn\" (UID: \"60e3a171-27e7-4ae6-ad6c-89984ecf5634\") " pod="openshift-console/console-7698dd44-8ddgn" Nov 01 19:59:08.929372 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:08.928726 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/60e3a171-27e7-4ae6-ad6c-89984ecf5634-console-oauth-config\") pod \"console-7698dd44-8ddgn\" (UID: \"60e3a171-27e7-4ae6-ad6c-89984ecf5634\") " pod="openshift-console/console-7698dd44-8ddgn" Nov 01 19:59:08.929372 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:08.928770 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/60e3a171-27e7-4ae6-ad6c-89984ecf5634-console-config\") pod \"console-7698dd44-8ddgn\" (UID: \"60e3a171-27e7-4ae6-ad6c-89984ecf5634\") " pod="openshift-console/console-7698dd44-8ddgn" Nov 01 19:59:08.929372 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:08.928802 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60e3a171-27e7-4ae6-ad6c-89984ecf5634-trusted-ca-bundle\") pod \"console-7698dd44-8ddgn\" (UID: \"60e3a171-27e7-4ae6-ad6c-89984ecf5634\") " pod="openshift-console/console-7698dd44-8ddgn" Nov 01 19:59:08.929372 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:08.928832 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/60e3a171-27e7-4ae6-ad6c-89984ecf5634-oauth-serving-cert\") pod \"console-7698dd44-8ddgn\" (UID: \"60e3a171-27e7-4ae6-ad6c-89984ecf5634\") " pod="openshift-console/console-7698dd44-8ddgn" Nov 01 19:59:08.929372 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:08.928879 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/60e3a171-27e7-4ae6-ad6c-89984ecf5634-service-ca\") pod \"console-7698dd44-8ddgn\" (UID: \"60e3a171-27e7-4ae6-ad6c-89984ecf5634\") " pod="openshift-console/console-7698dd44-8ddgn" Nov 01 19:59:08.929372 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:08.928908 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/60e3a171-27e7-4ae6-ad6c-89984ecf5634-tmp\") pod \"console-7698dd44-8ddgn\" (UID: \"60e3a171-27e7-4ae6-ad6c-89984ecf5634\") " pod="openshift-console/console-7698dd44-8ddgn" Nov 01 19:59:08.929372 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:08.928968 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/60e3a171-27e7-4ae6-ad6c-89984ecf5634-console-serving-cert\") pod \"console-7698dd44-8ddgn\" (UID: \"60e3a171-27e7-4ae6-ad6c-89984ecf5634\") " pod="openshift-console/console-7698dd44-8ddgn" Nov 01 19:59:08.929372 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:08.929009 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cq6h\" (UniqueName: \"kubernetes.io/projected/60e3a171-27e7-4ae6-ad6c-89984ecf5634-kube-api-access-4cq6h\") pod \"console-7698dd44-8ddgn\" (UID: \"60e3a171-27e7-4ae6-ad6c-89984ecf5634\") " pod="openshift-console/console-7698dd44-8ddgn" Nov 01 19:59:08.929845 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:08.929533 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/60e3a171-27e7-4ae6-ad6c-89984ecf5634-console-config\") pod \"console-7698dd44-8ddgn\" (UID: \"60e3a171-27e7-4ae6-ad6c-89984ecf5634\") " pod="openshift-console/console-7698dd44-8ddgn" Nov 01 19:59:08.929845 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:08.929674 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/60e3a171-27e7-4ae6-ad6c-89984ecf5634-tmp\") pod \"console-7698dd44-8ddgn\" (UID: \"60e3a171-27e7-4ae6-ad6c-89984ecf5634\") " pod="openshift-console/console-7698dd44-8ddgn" Nov 01 19:59:08.929845 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:08.929784 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/60e3a171-27e7-4ae6-ad6c-89984ecf5634-service-ca\") pod \"console-7698dd44-8ddgn\" (UID: \"60e3a171-27e7-4ae6-ad6c-89984ecf5634\") " pod="openshift-console/console-7698dd44-8ddgn" Nov 01 19:59:08.930329 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:08.930263 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/60e3a171-27e7-4ae6-ad6c-89984ecf5634-oauth-serving-cert\") pod \"console-7698dd44-8ddgn\" (UID: \"60e3a171-27e7-4ae6-ad6c-89984ecf5634\") " pod="openshift-console/console-7698dd44-8ddgn" Nov 01 19:59:08.930626 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:08.930603 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60e3a171-27e7-4ae6-ad6c-89984ecf5634-trusted-ca-bundle\") pod \"console-7698dd44-8ddgn\" (UID: \"60e3a171-27e7-4ae6-ad6c-89984ecf5634\") " pod="openshift-console/console-7698dd44-8ddgn" Nov 01 19:59:08.931737 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:08.931706 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/60e3a171-27e7-4ae6-ad6c-89984ecf5634-console-oauth-config\") pod \"console-7698dd44-8ddgn\" (UID: \"60e3a171-27e7-4ae6-ad6c-89984ecf5634\") " pod="openshift-console/console-7698dd44-8ddgn" Nov 01 19:59:08.932821 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:08.932798 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/60e3a171-27e7-4ae6-ad6c-89984ecf5634-console-serving-cert\") pod \"console-7698dd44-8ddgn\" (UID: \"60e3a171-27e7-4ae6-ad6c-89984ecf5634\") " pod="openshift-console/console-7698dd44-8ddgn" Nov 01 19:59:08.946054 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:08.946022 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cq6h\" (UniqueName: \"kubernetes.io/projected/60e3a171-27e7-4ae6-ad6c-89984ecf5634-kube-api-access-4cq6h\") pod \"console-7698dd44-8ddgn\" (UID: \"60e3a171-27e7-4ae6-ad6c-89984ecf5634\") " pod="openshift-console/console-7698dd44-8ddgn" Nov 01 19:59:08.956196 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:08.956173 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-596fbdd49c-2nmrq" Nov 01 19:59:09.031935 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:09.031731 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7698dd44-8ddgn" Nov 01 19:59:09.321172 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:09.321106 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7698dd44-8ddgn"] Nov 01 19:59:09.324273 ip-10-0-136-26 kubenswrapper[2576]: W1101 19:59:09.324246 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60e3a171_27e7_4ae6_ad6c_89984ecf5634.slice/crio-b3ea44655cb46e9c5ef929ad76a73cb9490badcb9c5405bde8f151cfa4aeb8c8 WatchSource:0}: Error finding container b3ea44655cb46e9c5ef929ad76a73cb9490badcb9c5405bde8f151cfa4aeb8c8: Status 404 returned error can't find the container with id b3ea44655cb46e9c5ef929ad76a73cb9490badcb9c5405bde8f151cfa4aeb8c8 Nov 01 19:59:09.985015 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:09.984973 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bvqvw" event={"ID":"65e6a200-c6f4-4671-9d2e-b0158ff3c5ce","Type":"ContainerStarted","Data":"db02babd3b8d67ba79721cb1fb469e5fa8c0c8e9800ae459216b30ef248d47a9"} Nov 01 19:59:09.985544 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:09.985074 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bvqvw" event={"ID":"65e6a200-c6f4-4671-9d2e-b0158ff3c5ce","Type":"ContainerStarted","Data":"edcca13e795b565f5af03ab7b89dd8c2e9a29c5b40e57ac344e535e8adbd08bf"} Nov 01 19:59:09.993345 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:09.992803 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7698dd44-8ddgn" event={"ID":"60e3a171-27e7-4ae6-ad6c-89984ecf5634","Type":"ContainerStarted","Data":"deab280cd317ce9dd53293758118ab4fe5e903b9ba385bc1baae6fac7a5a1a37"} Nov 01 19:59:09.993345 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:09.992852 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7698dd44-8ddgn" event={"ID":"60e3a171-27e7-4ae6-ad6c-89984ecf5634","Type":"ContainerStarted","Data":"b3ea44655cb46e9c5ef929ad76a73cb9490badcb9c5405bde8f151cfa4aeb8c8"} Nov 01 19:59:10.032140 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:10.032083 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7698dd44-8ddgn" podStartSLOduration=2.032064958 podStartE2EDuration="2.032064958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 19:59:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:59:10.0307048 +0000 UTC m=+68.052431091" watchObservedRunningTime="2026-11-01 19:59:10.032064958 +0000 UTC m=+68.053791247" Nov 01 19:59:10.032535 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:10.032492 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-bvqvw" podStartSLOduration=66.918558844 podStartE2EDuration="1m8.032484023s" totalImagesPullingTime="1.113925179s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 19:58:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:59:10.00347685 +0000 UTC m=+68.025203139" watchObservedRunningTime="2026-11-01 19:59:10.032484023 +0000 UTC m=+68.054210310" Nov 01 19:59:13.008429 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:13.008391 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"80c66ec7-5430-41a1-8641-56db5e4e9d73","Type":"ContainerStarted","Data":"517de8fdee23b80434e71946162632465d7c9580a7b69ade708ffdf401cdd144"} Nov 01 19:59:13.008854 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:13.008471 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"80c66ec7-5430-41a1-8641-56db5e4e9d73","Type":"ContainerStarted","Data":"e25643c77686e3395636e0b4a10f7904a73836d08aeba79db39bfad411bb3f0d"} Nov 01 19:59:13.008854 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:13.008510 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"80c66ec7-5430-41a1-8641-56db5e4e9d73","Type":"ContainerStarted","Data":"39ca1779d3e2325f0cc084bff04614b589b1f8e0c119fd669272dbd3c28dd02c"} Nov 01 19:59:13.008854 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:13.008540 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"80c66ec7-5430-41a1-8641-56db5e4e9d73","Type":"ContainerStarted","Data":"c51258636e3bb432293783857dddfce6b61d33076f8dcaf32df54d4d9da2f10e"} Nov 01 19:59:13.010451 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:13.010413 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-b7mcz" event={"ID":"16e51d04-90d5-484d-9f85-e9d484205220","Type":"ContainerStarted","Data":"56e546669d0868f94adac921de90905df54da4eb283263bd1b7cd6211a85e556"} Nov 01 19:59:13.011177 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:13.011155 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-b7mcz" Nov 01 19:59:13.026749 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:13.026701 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-b7mcz" podStartSLOduration=66.011534451 podStartE2EDuration="1m11.026685286s" totalImagesPullingTime="5.015150835s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 19:58:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:59:13.026605831 +0000 UTC m=+71.048332119" watchObservedRunningTime="2026-11-01 19:59:13.026685286 +0000 UTC m=+71.048411574" Nov 01 19:59:14.017389 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:14.017354 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"80c66ec7-5430-41a1-8641-56db5e4e9d73","Type":"ContainerStarted","Data":"54513e5b3e839c8bdba0581043ed5eac858babb463bc57d6f83f597d437a96a2"} Nov 01 19:59:14.017806 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:14.017474 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"80c66ec7-5430-41a1-8641-56db5e4e9d73","Type":"ContainerStarted","Data":"24713a72c5a0df1f0334a2d435e98bdb64cedb75a2385deb57868ffbb5c8f878"} Nov 01 19:59:14.018929 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:14.018903 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-k5brd" event={"ID":"2efc5bbf-efd4-4a42-bed8-53f2490bb7bc","Type":"ContainerStarted","Data":"26f9b2f9b750a0626b15e70144eb73c97598eb766b14d996be546be91a9e6f6b"} Nov 01 19:59:14.049584 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:14.049526 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.242659561 podStartE2EDuration="12.049513355s" totalImagesPullingTime="8.791624526s" totalInitContainerRuntime="15.229268ms" isStatefulPod=false podCreationTimestamp="2026-11-01 19:59:02 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:59:14.047549888 +0000 UTC m=+72.069276171" watchObservedRunningTime="2026-11-01 19:59:14.049513355 +0000 UTC m=+72.071239640" Nov 01 19:59:14.062492 ip-10-0-136-26 kubenswrapper[2576]: I1101 19:59:14.062451 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-k5brd" podStartSLOduration=65.914416796 podStartE2EDuration="1m12.062439439s" totalImagesPullingTime="6.148022643s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 19:58:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:59:14.06130126 +0000 UTC m=+72.083027537" watchObservedRunningTime="2026-11-01 19:59:14.062439439 +0000 UTC m=+72.084165750" Nov 01 19:59:14.161962 ip-10-0-136-26 systemd[1]: Stopping Kubernetes Kubelet... Nov 01 19:59:14.179813 ip-10-0-136-26 systemd[1]: kubelet.service: Deactivated successfully. Nov 01 19:59:14.179986 ip-10-0-136-26 systemd[1]: Stopped Kubernetes Kubelet. Nov 01 19:59:14.180039 ip-10-0-136-26 systemd[1]: kubelet.service: Consumed 3.810s CPU time. Nov 01 19:59:14.196118 ip-10-0-136-26 systemd[1]: Starting Kubernetes Kubelet... Nov 01 19:59:14.259686 ip-10-0-136-26 kubenswrapper[5935]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 19:59:14.259686 ip-10-0-136-26 kubenswrapper[5935]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 01 19:59:14.259686 ip-10-0-136-26 kubenswrapper[5935]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 19:59:14.259686 ip-10-0-136-26 kubenswrapper[5935]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 01 19:59:14.262085 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.261997 5935 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 19:59:14.262085 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262079 5935 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 19:59:14.262085 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262084 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 19:59:14.262085 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262088 5935 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 19:59:14.262380 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262093 5935 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 19:59:14.262380 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262097 5935 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 19:59:14.262380 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262101 5935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 19:59:14.262380 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262106 5935 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 19:59:14.262380 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262110 5935 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 19:59:14.262380 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262114 5935 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 19:59:14.262380 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262118 5935 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 19:59:14.262380 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262121 5935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 19:59:14.262380 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262125 5935 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 19:59:14.262380 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262129 5935 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 19:59:14.262380 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262133 5935 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 19:59:14.262380 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262138 5935 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 19:59:14.262380 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262141 5935 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 19:59:14.262380 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262145 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 19:59:14.262380 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262149 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 19:59:14.262380 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262153 5935 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 19:59:14.262380 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262157 5935 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 19:59:14.262380 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262160 5935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 19:59:14.262380 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262172 5935 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 19:59:14.263198 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262177 5935 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 19:59:14.263198 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262181 5935 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 19:59:14.263198 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262185 5935 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 19:59:14.263198 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262189 5935 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 19:59:14.263198 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262193 5935 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 19:59:14.263198 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262197 5935 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 19:59:14.263198 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262201 5935 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 19:59:14.263198 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262205 5935 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 19:59:14.263198 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262209 5935 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 19:59:14.263198 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262214 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 19:59:14.263198 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262218 5935 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 19:59:14.263198 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262223 5935 feature_gate.go:386] unrecognized feature gate: Example Nov 01 19:59:14.263198 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262227 5935 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 19:59:14.263198 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262231 5935 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 19:59:14.263198 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262236 5935 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 19:59:14.263198 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262240 5935 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 19:59:14.263198 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262244 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 19:59:14.263198 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262249 5935 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 19:59:14.263198 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262253 5935 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 19:59:14.263198 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262257 5935 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 19:59:14.264082 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262261 5935 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 19:59:14.264082 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262266 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 19:59:14.264082 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262270 5935 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 19:59:14.264082 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262275 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 19:59:14.264082 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262279 5935 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 19:59:14.264082 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262300 5935 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 19:59:14.264082 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262305 5935 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 19:59:14.264082 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262310 5935 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 19:59:14.264082 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262314 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 19:59:14.264082 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262318 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 19:59:14.264082 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262322 5935 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 19:59:14.264082 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262327 5935 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 19:59:14.264082 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262332 5935 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 19:59:14.264082 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262337 5935 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 19:59:14.264082 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262340 5935 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 19:59:14.264082 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262345 5935 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 19:59:14.264082 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262350 5935 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 19:59:14.264082 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262354 5935 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 19:59:14.264082 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262358 5935 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 19:59:14.264082 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262362 5935 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 19:59:14.264986 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262367 5935 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 19:59:14.264986 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262372 5935 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 19:59:14.264986 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262376 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 19:59:14.264986 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262381 5935 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 19:59:14.264986 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262385 5935 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 19:59:14.264986 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262389 5935 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 19:59:14.264986 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262393 5935 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 19:59:14.264986 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262397 5935 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 19:59:14.264986 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262404 5935 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 19:59:14.264986 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262414 5935 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 19:59:14.264986 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262419 5935 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 19:59:14.264986 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262424 5935 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 19:59:14.264986 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262428 5935 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 19:59:14.264986 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262432 5935 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 19:59:14.264986 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262436 5935 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 19:59:14.264986 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262440 5935 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 19:59:14.264986 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262445 5935 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 19:59:14.264986 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262448 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 19:59:14.264986 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262453 5935 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 19:59:14.264986 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262457 5935 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 19:59:14.265970 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262461 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 19:59:14.265970 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262466 5935 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 19:59:14.265970 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262472 5935 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 19:59:14.265970 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262479 5935 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 19:59:14.265970 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262486 5935 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 19:59:14.265970 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262490 5935 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 19:59:14.265970 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262497 5935 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 19:59:14.265970 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262503 5935 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 19:59:14.265970 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262509 5935 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 19:59:14.265970 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262514 5935 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 19:59:14.265970 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262520 5935 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 19:59:14.265970 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262525 5935 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 19:59:14.265970 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262529 5935 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 19:59:14.265970 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262534 5935 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 19:59:14.265970 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262538 5935 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 19:59:14.265970 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262543 5935 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 19:59:14.265970 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262548 5935 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 19:59:14.265970 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262552 5935 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 19:59:14.266803 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262556 5935 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 19:59:14.266803 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262560 5935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 19:59:14.266803 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262568 5935 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 19:59:14.266803 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262572 5935 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 19:59:14.266803 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.262577 5935 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 19:59:14.266803 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.263831 5935 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 19:59:14.266803 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.263841 5935 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 19:59:14.266803 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.263846 5935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 19:59:14.266803 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.263851 5935 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 19:59:14.266803 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.263855 5935 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 19:59:14.266803 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.263859 5935 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 19:59:14.266803 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.263863 5935 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 19:59:14.266803 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.263868 5935 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 19:59:14.266803 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.263872 5935 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 19:59:14.266803 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.263877 5935 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 19:59:14.266803 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.263881 5935 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 19:59:14.266803 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.263886 5935 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 19:59:14.266803 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.263890 5935 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 19:59:14.266803 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.263896 5935 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 19:59:14.267750 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.263900 5935 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 19:59:14.267750 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.263904 5935 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 19:59:14.267750 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.263908 5935 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 19:59:14.267750 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.263913 5935 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 19:59:14.267750 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.263917 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 19:59:14.267750 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.263921 5935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 19:59:14.267750 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.263926 5935 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 19:59:14.267750 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.263930 5935 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 19:59:14.267750 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.263935 5935 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 19:59:14.267750 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.263940 5935 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 19:59:14.267750 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.263944 5935 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 19:59:14.267750 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.263948 5935 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 19:59:14.267750 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.263952 5935 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 19:59:14.267750 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.263956 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 19:59:14.267750 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.263961 5935 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 19:59:14.267750 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.263965 5935 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 19:59:14.267750 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.263970 5935 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 19:59:14.267750 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.263974 5935 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 19:59:14.267750 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.263979 5935 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 19:59:14.267750 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.263984 5935 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 19:59:14.268630 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.263991 5935 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 19:59:14.268630 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.263998 5935 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 19:59:14.268630 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.264002 5935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 19:59:14.268630 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.264007 5935 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 19:59:14.268630 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.264011 5935 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 19:59:14.268630 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.264016 5935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 19:59:14.268630 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.264020 5935 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 19:59:14.268630 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.264024 5935 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 19:59:14.268630 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.264029 5935 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 19:59:14.268630 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.264035 5935 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 19:59:14.268630 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.264041 5935 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 19:59:14.268630 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.264046 5935 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 19:59:14.268630 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.264050 5935 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 19:59:14.268630 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.264056 5935 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 19:59:14.268630 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.264060 5935 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 19:59:14.268630 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.264065 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 19:59:14.268630 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.264069 5935 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 19:59:14.268630 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.264074 5935 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 19:59:14.268630 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.264078 5935 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 19:59:14.269144 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.264082 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 19:59:14.269144 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.264086 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 19:59:14.269144 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.264090 5935 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 19:59:14.269144 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.264094 5935 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 19:59:14.269144 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.264098 5935 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 19:59:14.269144 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.264102 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 19:59:14.269144 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.264107 5935 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 19:59:14.269144 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.264111 5935 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 19:59:14.269144 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.264116 5935 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 19:59:14.269144 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.264120 5935 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 19:59:14.269144 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.264124 5935 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 19:59:14.269144 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.264129 5935 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 19:59:14.269144 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.264134 5935 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 19:59:14.269144 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.264138 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 19:59:14.269144 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.264143 5935 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 19:59:14.269144 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.264147 5935 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 19:59:14.269144 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.264151 5935 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 19:59:14.269144 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.264155 5935 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 19:59:14.269144 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.264159 5935 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 19:59:14.269144 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.264163 5935 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 19:59:14.269705 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.264168 5935 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 19:59:14.269705 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.264172 5935 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 19:59:14.269705 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.264177 5935 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 19:59:14.269705 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.264181 5935 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 19:59:14.269705 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.264186 5935 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 19:59:14.269705 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.264190 5935 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 19:59:14.269705 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.264194 5935 feature_gate.go:386] unrecognized feature gate: Example Nov 01 19:59:14.269705 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.264198 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 19:59:14.269705 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.264202 5935 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 19:59:14.269705 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.264206 5935 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 19:59:14.269705 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.264210 5935 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 19:59:14.269705 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.264215 5935 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 19:59:14.269705 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.264220 5935 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 19:59:14.269705 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.264224 5935 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 19:59:14.269705 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.264229 5935 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 19:59:14.269705 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.264233 5935 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 19:59:14.269705 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.264237 5935 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 19:59:14.269705 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.264241 5935 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 19:59:14.269705 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.264246 5935 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 19:59:14.270192 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.264250 5935 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 19:59:14.270192 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.264254 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 19:59:14.270192 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.264258 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 19:59:14.270192 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.264263 5935 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 19:59:14.270192 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.264267 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 19:59:14.270192 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.264271 5935 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 19:59:14.270192 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.264276 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 19:59:14.270192 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.264296 5935 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 19:59:14.270192 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.264302 5935 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 19:59:14.270192 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.264306 5935 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 19:59:14.270192 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.264310 5935 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 19:59:14.270192 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.264315 5935 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 19:59:14.270192 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.264321 5935 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 19:59:14.270192 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.264325 5935 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 19:59:14.270192 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265142 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 19:59:14.270192 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265151 5935 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 19:59:14.270192 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265156 5935 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 19:59:14.270192 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265161 5935 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 19:59:14.270192 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265166 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 19:59:14.270725 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265171 5935 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 19:59:14.270725 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265175 5935 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 19:59:14.270725 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265229 5935 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 19:59:14.270725 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265235 5935 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 19:59:14.270725 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265240 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 19:59:14.270725 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265244 5935 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 19:59:14.270725 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265272 5935 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 19:59:14.270725 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265277 5935 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 19:59:14.270725 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265297 5935 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 19:59:14.270725 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265302 5935 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 19:59:14.270725 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265306 5935 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 19:59:14.270725 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265311 5935 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 19:59:14.270725 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265315 5935 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 19:59:14.270725 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265319 5935 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 19:59:14.270725 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265323 5935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 19:59:14.270725 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265327 5935 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 19:59:14.270725 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265331 5935 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 19:59:14.270725 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265336 5935 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 19:59:14.270725 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265340 5935 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 19:59:14.270725 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265353 5935 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 19:59:14.271213 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265358 5935 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 19:59:14.271213 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265362 5935 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 19:59:14.271213 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265367 5935 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 19:59:14.271213 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265371 5935 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 19:59:14.271213 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265375 5935 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 19:59:14.271213 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265379 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 19:59:14.271213 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265383 5935 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 19:59:14.271213 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265388 5935 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 19:59:14.271213 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265392 5935 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 19:59:14.271213 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265396 5935 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 19:59:14.271213 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265402 5935 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 19:59:14.271213 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265406 5935 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 19:59:14.271213 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265411 5935 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 19:59:14.271213 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265415 5935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 19:59:14.271213 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265419 5935 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 19:59:14.271213 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265423 5935 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 19:59:14.271213 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265427 5935 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 19:59:14.271213 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265432 5935 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 19:59:14.271213 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265436 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 19:59:14.271701 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265440 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 19:59:14.271701 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265446 5935 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 19:59:14.271701 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265450 5935 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 19:59:14.271701 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265455 5935 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 19:59:14.271701 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265459 5935 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 19:59:14.271701 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265463 5935 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 19:59:14.271701 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265467 5935 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 19:59:14.271701 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265471 5935 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 19:59:14.271701 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265476 5935 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 19:59:14.271701 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265480 5935 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 19:59:14.271701 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265484 5935 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 19:59:14.271701 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265488 5935 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 19:59:14.271701 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265492 5935 feature_gate.go:386] unrecognized feature gate: Example Nov 01 19:59:14.271701 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265497 5935 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 19:59:14.271701 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265502 5935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 19:59:14.271701 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265506 5935 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 19:59:14.271701 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265510 5935 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 19:59:14.271701 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265515 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 19:59:14.271701 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265519 5935 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 19:59:14.271701 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265531 5935 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 19:59:14.272212 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265538 5935 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 19:59:14.272212 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265543 5935 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 19:59:14.272212 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265547 5935 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 19:59:14.272212 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265552 5935 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 19:59:14.272212 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265557 5935 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 19:59:14.272212 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265561 5935 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 19:59:14.272212 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265565 5935 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 19:59:14.272212 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265569 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 19:59:14.272212 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265574 5935 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 19:59:14.272212 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265580 5935 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 19:59:14.272212 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265584 5935 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 19:59:14.272212 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265589 5935 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 19:59:14.272212 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265593 5935 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 19:59:14.272212 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265599 5935 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 19:59:14.272212 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265605 5935 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 19:59:14.272212 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265610 5935 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 19:59:14.272212 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265615 5935 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 19:59:14.272212 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265619 5935 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 19:59:14.272212 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265623 5935 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 19:59:14.272922 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265627 5935 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 19:59:14.272922 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265631 5935 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 19:59:14.272922 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265636 5935 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 19:59:14.272922 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265640 5935 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 19:59:14.272922 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265645 5935 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 19:59:14.272922 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265649 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 19:59:14.272922 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265653 5935 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 19:59:14.272922 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265667 5935 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 19:59:14.272922 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265671 5935 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 19:59:14.272922 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265678 5935 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 19:59:14.272922 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265684 5935 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 19:59:14.272922 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265690 5935 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 19:59:14.272922 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265694 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 19:59:14.272922 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265700 5935 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 19:59:14.272922 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265705 5935 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 19:59:14.272922 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265710 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 19:59:14.272922 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265714 5935 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 19:59:14.272922 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265719 5935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 19:59:14.272922 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265724 5935 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 19:59:14.274242 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265728 5935 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 19:59:14.274242 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265732 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 19:59:14.274242 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265737 5935 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 19:59:14.274242 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.265741 5935 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 19:59:14.274242 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.265882 5935 flags.go:64] FLAG: --address="0.0.0.0" Nov 01 19:59:14.274242 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.265894 5935 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 01 19:59:14.274242 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.265910 5935 flags.go:64] FLAG: --anonymous-auth="true" Nov 01 19:59:14.274242 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.265917 5935 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 01 19:59:14.274242 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.265924 5935 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 01 19:59:14.274242 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.265929 5935 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 01 19:59:14.274242 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.265936 5935 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 01 19:59:14.274242 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.265943 5935 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 01 19:59:14.274242 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.265948 5935 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 01 19:59:14.274242 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.265953 5935 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 01 19:59:14.274242 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.265959 5935 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 01 19:59:14.274242 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.265964 5935 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 01 19:59:14.274242 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.265969 5935 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 01 19:59:14.274242 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.265973 5935 flags.go:64] FLAG: --cgroup-root="" Nov 01 19:59:14.274242 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.265978 5935 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 01 19:59:14.274242 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.265982 5935 flags.go:64] FLAG: --client-ca-file="" Nov 01 19:59:14.274242 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.265987 5935 flags.go:64] FLAG: --cloud-provider="external" Nov 01 19:59:14.274242 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.265993 5935 flags.go:64] FLAG: --cluster-dns="[]" Nov 01 19:59:14.274242 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.265998 5935 flags.go:64] FLAG: --cluster-domain="" Nov 01 19:59:14.274242 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266003 5935 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 01 19:59:14.274859 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266009 5935 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 01 19:59:14.274859 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266014 5935 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 01 19:59:14.274859 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266021 5935 flags.go:64] FLAG: --container-log-max-files="5" Nov 01 19:59:14.274859 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266028 5935 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 01 19:59:14.274859 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266033 5935 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 01 19:59:14.274859 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266038 5935 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 01 19:59:14.274859 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266043 5935 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 01 19:59:14.274859 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266048 5935 flags.go:64] FLAG: --contention-profiling="false" Nov 01 19:59:14.274859 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266053 5935 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 01 19:59:14.274859 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266058 5935 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 01 19:59:14.274859 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266063 5935 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 01 19:59:14.274859 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266068 5935 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 01 19:59:14.274859 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266075 5935 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 01 19:59:14.274859 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266080 5935 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 01 19:59:14.274859 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266084 5935 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 01 19:59:14.274859 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266115 5935 flags.go:64] FLAG: --enable-load-reader="false" Nov 01 19:59:14.274859 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266121 5935 flags.go:64] FLAG: --enable-server="true" Nov 01 19:59:14.274859 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266126 5935 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 01 19:59:14.274859 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266135 5935 flags.go:64] FLAG: --event-burst="100" Nov 01 19:59:14.274859 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266140 5935 flags.go:64] FLAG: --event-qps="50" Nov 01 19:59:14.274859 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266145 5935 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 01 19:59:14.274859 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266150 5935 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 01 19:59:14.274859 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266155 5935 flags.go:64] FLAG: --eviction-hard="" Nov 01 19:59:14.274859 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266164 5935 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 01 19:59:14.275500 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266169 5935 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 01 19:59:14.275500 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266174 5935 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 01 19:59:14.275500 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266179 5935 flags.go:64] FLAG: --eviction-soft="" Nov 01 19:59:14.275500 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266184 5935 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 01 19:59:14.275500 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266189 5935 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 01 19:59:14.275500 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266194 5935 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 01 19:59:14.275500 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266200 5935 flags.go:64] FLAG: --experimental-mounter-path="" Nov 01 19:59:14.275500 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266205 5935 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 01 19:59:14.275500 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266210 5935 flags.go:64] FLAG: --fail-swap-on="true" Nov 01 19:59:14.275500 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266215 5935 flags.go:64] FLAG: --feature-gates="" Nov 01 19:59:14.275500 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266223 5935 flags.go:64] FLAG: --file-check-frequency="20s" Nov 01 19:59:14.275500 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266228 5935 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 01 19:59:14.275500 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266234 5935 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 01 19:59:14.275500 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266239 5935 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 01 19:59:14.275500 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266244 5935 flags.go:64] FLAG: --healthz-port="10248" Nov 01 19:59:14.275500 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266249 5935 flags.go:64] FLAG: --help="false" Nov 01 19:59:14.275500 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266253 5935 flags.go:64] FLAG: --hostname-override="ip-10-0-136-26.us-east-2.compute.internal" Nov 01 19:59:14.275500 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266259 5935 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 01 19:59:14.275500 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266264 5935 flags.go:64] FLAG: --http-check-frequency="20s" Nov 01 19:59:14.275500 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266269 5935 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 01 19:59:14.275500 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266274 5935 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 01 19:59:14.275500 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266280 5935 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 01 19:59:14.275500 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266301 5935 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 01 19:59:14.276064 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266306 5935 flags.go:64] FLAG: --image-service-endpoint="" Nov 01 19:59:14.276064 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266311 5935 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 01 19:59:14.276064 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266315 5935 flags.go:64] FLAG: --kube-api-burst="100" Nov 01 19:59:14.276064 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266321 5935 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 01 19:59:14.276064 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266326 5935 flags.go:64] FLAG: --kube-api-qps="50" Nov 01 19:59:14.276064 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266330 5935 flags.go:64] FLAG: --kube-reserved="" Nov 01 19:59:14.276064 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266335 5935 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 01 19:59:14.276064 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266340 5935 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 01 19:59:14.276064 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266347 5935 flags.go:64] FLAG: --kubelet-cgroups="" Nov 01 19:59:14.276064 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266352 5935 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 01 19:59:14.276064 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266357 5935 flags.go:64] FLAG: --lock-file="" Nov 01 19:59:14.276064 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266361 5935 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 01 19:59:14.276064 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266366 5935 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 01 19:59:14.276064 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266371 5935 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 01 19:59:14.276064 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266380 5935 flags.go:64] FLAG: --log-json-split-stream="false" Nov 01 19:59:14.276064 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266385 5935 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 01 19:59:14.276064 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266391 5935 flags.go:64] FLAG: --log-text-split-stream="false" Nov 01 19:59:14.276064 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266396 5935 flags.go:64] FLAG: --logging-format="text" Nov 01 19:59:14.276064 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266400 5935 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 01 19:59:14.276064 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266408 5935 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 01 19:59:14.276064 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266412 5935 flags.go:64] FLAG: --manifest-url="" Nov 01 19:59:14.276064 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266418 5935 flags.go:64] FLAG: --manifest-url-header="" Nov 01 19:59:14.276064 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266425 5935 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 01 19:59:14.276064 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266430 5935 flags.go:64] FLAG: --max-open-files="1000000" Nov 01 19:59:14.276064 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266441 5935 flags.go:64] FLAG: --max-pods="110" Nov 01 19:59:14.276704 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266447 5935 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 01 19:59:14.276704 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266452 5935 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 01 19:59:14.276704 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266456 5935 flags.go:64] FLAG: --memory-manager-policy="None" Nov 01 19:59:14.276704 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266461 5935 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 01 19:59:14.276704 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266466 5935 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 01 19:59:14.276704 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266471 5935 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 01 19:59:14.276704 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266476 5935 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 01 19:59:14.276704 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266487 5935 flags.go:64] FLAG: --node-status-max-images="50" Nov 01 19:59:14.276704 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266492 5935 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 01 19:59:14.276704 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266496 5935 flags.go:64] FLAG: --oom-score-adj="-999" Nov 01 19:59:14.276704 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266501 5935 flags.go:64] FLAG: --pod-cidr="" Nov 01 19:59:14.276704 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266506 5935 flags.go:64] FLAG: --pod-manifest-path="" Nov 01 19:59:14.276704 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266511 5935 flags.go:64] FLAG: --pod-max-pids="-1" Nov 01 19:59:14.276704 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266516 5935 flags.go:64] FLAG: --pods-per-core="0" Nov 01 19:59:14.276704 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266521 5935 flags.go:64] FLAG: --port="10250" Nov 01 19:59:14.276704 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266528 5935 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 01 19:59:14.276704 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266533 5935 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-04d4951a5c963ee10" Nov 01 19:59:14.276704 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266538 5935 flags.go:64] FLAG: --qos-reserved="" Nov 01 19:59:14.276704 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266543 5935 flags.go:64] FLAG: --read-only-port="10255" Nov 01 19:59:14.276704 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266549 5935 flags.go:64] FLAG: --register-node="true" Nov 01 19:59:14.276704 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266553 5935 flags.go:64] FLAG: --register-with-taints="" Nov 01 19:59:14.276704 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266560 5935 flags.go:64] FLAG: --registry-burst="10" Nov 01 19:59:14.276704 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266565 5935 flags.go:64] FLAG: --registry-qps="5" Nov 01 19:59:14.276704 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266570 5935 flags.go:64] FLAG: --reserved-cpus="" Nov 01 19:59:14.276704 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266575 5935 flags.go:64] FLAG: --reserved-memory="" Nov 01 19:59:14.276704 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266581 5935 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 01 19:59:14.277342 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266588 5935 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 01 19:59:14.277342 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266593 5935 flags.go:64] FLAG: --rotate-certificates="false" Nov 01 19:59:14.277342 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266598 5935 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 01 19:59:14.277342 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266603 5935 flags.go:64] FLAG: --runonce="false" Nov 01 19:59:14.277342 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266608 5935 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 01 19:59:14.277342 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266613 5935 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 01 19:59:14.277342 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266619 5935 flags.go:64] FLAG: --seccomp-default="false" Nov 01 19:59:14.277342 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266623 5935 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 01 19:59:14.277342 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266628 5935 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 01 19:59:14.277342 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266633 5935 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 01 19:59:14.277342 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266638 5935 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 01 19:59:14.277342 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266643 5935 flags.go:64] FLAG: --storage-driver-password="root" Nov 01 19:59:14.277342 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266648 5935 flags.go:64] FLAG: --storage-driver-secure="false" Nov 01 19:59:14.277342 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266653 5935 flags.go:64] FLAG: --storage-driver-table="stats" Nov 01 19:59:14.277342 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266657 5935 flags.go:64] FLAG: --storage-driver-user="root" Nov 01 19:59:14.277342 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266662 5935 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 01 19:59:14.277342 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266667 5935 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 01 19:59:14.277342 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266672 5935 flags.go:64] FLAG: --system-cgroups="" Nov 01 19:59:14.277342 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266676 5935 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 01 19:59:14.277342 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266685 5935 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 01 19:59:14.277342 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266691 5935 flags.go:64] FLAG: --tls-cert-file="" Nov 01 19:59:14.277342 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266698 5935 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 01 19:59:14.277342 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266704 5935 flags.go:64] FLAG: --tls-min-version="" Nov 01 19:59:14.277342 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266709 5935 flags.go:64] FLAG: --tls-private-key-file="" Nov 01 19:59:14.277342 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266714 5935 flags.go:64] FLAG: --topology-manager-policy="none" Nov 01 19:59:14.278062 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266718 5935 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 01 19:59:14.278062 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266723 5935 flags.go:64] FLAG: --topology-manager-scope="container" Nov 01 19:59:14.278062 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266728 5935 flags.go:64] FLAG: --v="2" Nov 01 19:59:14.278062 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266735 5935 flags.go:64] FLAG: --version="false" Nov 01 19:59:14.278062 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266741 5935 flags.go:64] FLAG: --vmodule="" Nov 01 19:59:14.278062 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266747 5935 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 01 19:59:14.278062 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.266753 5935 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 01 19:59:14.278062 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.266990 5935 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 19:59:14.278062 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.266999 5935 feature_gate.go:386] unrecognized feature gate: Example Nov 01 19:59:14.278062 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267006 5935 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 19:59:14.278062 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267012 5935 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 19:59:14.278062 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267017 5935 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 19:59:14.278062 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267022 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 19:59:14.278062 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267026 5935 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 19:59:14.278062 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267031 5935 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 19:59:14.278062 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267035 5935 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 19:59:14.278062 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267039 5935 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 19:59:14.278062 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267043 5935 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 19:59:14.278062 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267048 5935 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 19:59:14.278062 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267052 5935 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 19:59:14.278062 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267056 5935 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 19:59:14.278669 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267061 5935 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 19:59:14.278669 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267065 5935 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 19:59:14.278669 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267070 5935 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 19:59:14.278669 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267074 5935 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 19:59:14.278669 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267079 5935 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 19:59:14.278669 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267083 5935 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 19:59:14.278669 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267087 5935 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 19:59:14.278669 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267094 5935 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 19:59:14.278669 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267099 5935 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 19:59:14.278669 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267103 5935 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 19:59:14.278669 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267107 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 19:59:14.278669 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267112 5935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 19:59:14.278669 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267116 5935 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 19:59:14.278669 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267120 5935 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 19:59:14.278669 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267126 5935 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 19:59:14.278669 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267132 5935 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 19:59:14.278669 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267136 5935 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 19:59:14.278669 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267141 5935 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 19:59:14.278669 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267148 5935 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 19:59:14.279190 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267152 5935 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 19:59:14.279190 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267156 5935 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 19:59:14.279190 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267160 5935 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 19:59:14.279190 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267165 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 19:59:14.279190 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267169 5935 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 19:59:14.279190 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267174 5935 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 19:59:14.279190 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267178 5935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 19:59:14.279190 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267183 5935 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 19:59:14.279190 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267187 5935 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 19:59:14.279190 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267191 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 19:59:14.279190 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267196 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 19:59:14.279190 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267200 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 19:59:14.279190 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267205 5935 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 19:59:14.279190 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267209 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 19:59:14.279190 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267214 5935 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 19:59:14.279190 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267218 5935 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 19:59:14.279190 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267222 5935 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 19:59:14.279190 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267227 5935 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 19:59:14.279190 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267231 5935 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 19:59:14.279190 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267235 5935 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 19:59:14.279828 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267241 5935 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 19:59:14.279828 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267246 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 19:59:14.279828 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267250 5935 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 19:59:14.279828 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267254 5935 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 19:59:14.279828 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267259 5935 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 19:59:14.279828 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267263 5935 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 19:59:14.279828 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267267 5935 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 19:59:14.279828 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267272 5935 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 19:59:14.279828 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267276 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 19:59:14.279828 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267280 5935 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 19:59:14.279828 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267300 5935 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 19:59:14.279828 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267307 5935 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 19:59:14.279828 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267312 5935 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 19:59:14.279828 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267316 5935 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 19:59:14.279828 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267321 5935 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 19:59:14.279828 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267325 5935 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 19:59:14.279828 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267330 5935 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 19:59:14.279828 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267335 5935 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 19:59:14.279828 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267339 5935 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 19:59:14.280586 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267343 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 19:59:14.280586 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267348 5935 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 19:59:14.280586 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267352 5935 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 19:59:14.280586 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267356 5935 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 19:59:14.280586 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267360 5935 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 19:59:14.280586 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267365 5935 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 19:59:14.280586 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267369 5935 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 19:59:14.280586 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267373 5935 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 19:59:14.280586 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267378 5935 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 19:59:14.280586 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267382 5935 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 19:59:14.280586 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267386 5935 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 19:59:14.280586 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267391 5935 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 19:59:14.280586 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267395 5935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 19:59:14.280586 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267401 5935 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 19:59:14.280586 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267406 5935 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 19:59:14.280586 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267410 5935 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 19:59:14.280586 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267414 5935 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 19:59:14.280586 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267419 5935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 19:59:14.280586 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267423 5935 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 19:59:14.281305 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267427 5935 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 19:59:14.281305 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267431 5935 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 19:59:14.281305 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267435 5935 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 19:59:14.281305 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267440 5935 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 19:59:14.281305 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267444 5935 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 19:59:14.281305 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267450 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 19:59:14.281305 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267454 5935 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 19:59:14.281305 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267459 5935 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 19:59:14.281305 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267464 5935 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 19:59:14.281305 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267470 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 19:59:14.281305 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267474 5935 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 19:59:14.281305 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267479 5935 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 19:59:14.281305 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267484 5935 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 19:59:14.281305 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267489 5935 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 19:59:14.281305 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267493 5935 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 19:59:14.281305 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.267540 5935 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 19:59:14.282001 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267595 5935 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 19:59:14.282001 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267602 5935 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 19:59:14.282001 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267607 5935 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 19:59:14.282001 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267611 5935 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 19:59:14.282001 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267616 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 19:59:14.282001 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267620 5935 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 19:59:14.282001 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267625 5935 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 19:59:14.282001 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267629 5935 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 19:59:14.282001 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267634 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 19:59:14.282001 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267638 5935 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 19:59:14.282001 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267644 5935 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 19:59:14.282001 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267649 5935 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 19:59:14.282001 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267654 5935 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 19:59:14.282001 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267658 5935 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 19:59:14.282001 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267663 5935 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 19:59:14.282001 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267668 5935 feature_gate.go:386] unrecognized feature gate: Example Nov 01 19:59:14.282001 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267675 5935 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 19:59:14.282001 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267681 5935 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 19:59:14.282001 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267685 5935 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 19:59:14.282852 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267689 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 19:59:14.282852 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267693 5935 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 19:59:14.282852 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267698 5935 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 19:59:14.282852 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267704 5935 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 19:59:14.282852 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267708 5935 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 19:59:14.282852 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267713 5935 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 19:59:14.282852 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267718 5935 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 19:59:14.282852 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267722 5935 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 19:59:14.282852 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267726 5935 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 19:59:14.282852 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267731 5935 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 19:59:14.282852 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267735 5935 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 19:59:14.282852 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267740 5935 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 19:59:14.282852 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267744 5935 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 19:59:14.282852 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267748 5935 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 19:59:14.282852 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267752 5935 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 19:59:14.282852 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267756 5935 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 19:59:14.282852 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267760 5935 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 19:59:14.282852 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267764 5935 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 19:59:14.282852 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267768 5935 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 19:59:14.282852 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267772 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 19:59:14.283704 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267777 5935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 19:59:14.283704 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267781 5935 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 19:59:14.283704 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267786 5935 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 19:59:14.283704 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267798 5935 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 19:59:14.283704 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267804 5935 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 19:59:14.283704 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267810 5935 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 19:59:14.283704 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267815 5935 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 19:59:14.283704 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267819 5935 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 19:59:14.283704 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267823 5935 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 19:59:14.283704 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267827 5935 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 19:59:14.283704 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267832 5935 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 19:59:14.283704 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267836 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 19:59:14.283704 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267841 5935 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 19:59:14.283704 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267845 5935 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 19:59:14.283704 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267850 5935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 19:59:14.283704 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267856 5935 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 19:59:14.283704 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267860 5935 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 19:59:14.283704 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267865 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 19:59:14.283704 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267870 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 19:59:14.284541 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267875 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 19:59:14.284541 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267879 5935 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 19:59:14.284541 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267884 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 19:59:14.284541 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267888 5935 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 19:59:14.284541 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267892 5935 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 19:59:14.284541 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267897 5935 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 19:59:14.284541 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267901 5935 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 19:59:14.284541 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267905 5935 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 19:59:14.284541 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267910 5935 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 19:59:14.284541 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267914 5935 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 19:59:14.284541 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267919 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 19:59:14.284541 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267923 5935 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 19:59:14.284541 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267928 5935 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 19:59:14.284541 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267932 5935 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 19:59:14.284541 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267937 5935 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 19:59:14.284541 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267941 5935 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 19:59:14.284541 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267947 5935 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 19:59:14.284541 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267951 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 19:59:14.284541 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267956 5935 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 19:59:14.285346 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267960 5935 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 19:59:14.285346 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267966 5935 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 19:59:14.285346 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267970 5935 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 19:59:14.285346 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267974 5935 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 19:59:14.285346 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267978 5935 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 19:59:14.285346 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267983 5935 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 19:59:14.285346 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267987 5935 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 19:59:14.285346 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267992 5935 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 19:59:14.285346 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.267997 5935 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 19:59:14.285346 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.268004 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 19:59:14.285346 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.268008 5935 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 19:59:14.285346 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.268013 5935 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 19:59:14.285346 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.268017 5935 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 19:59:14.285346 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.268022 5935 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 19:59:14.285346 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.268026 5935 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 19:59:14.285346 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.268030 5935 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 19:59:14.285346 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.268035 5935 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 19:59:14.285346 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.268039 5935 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 19:59:14.285346 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.268043 5935 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 19:59:14.285346 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.268048 5935 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 19:59:14.286230 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.268052 5935 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 19:59:14.286230 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.268056 5935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 19:59:14.286230 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.268060 5935 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 19:59:14.286230 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.268065 5935 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 19:59:14.286230 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.268069 5935 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 19:59:14.286230 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.268073 5935 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 19:59:14.286230 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.268077 5935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 19:59:14.286230 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.268082 5935 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 19:59:14.286230 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.268086 5935 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 19:59:14.286230 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.271801 5935 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 01 19:59:14.286230 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.271816 5935 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 01 19:59:14.286230 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.271833 5935 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 01 19:59:14.286230 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.271838 5935 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 01 19:59:14.286230 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.271894 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 19:59:14.286230 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.271899 5935 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 19:59:14.286230 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.271902 5935 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 19:59:14.286230 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.271904 5935 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 19:59:14.286230 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.271907 5935 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 19:59:14.286230 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.271910 5935 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 19:59:14.287272 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.271913 5935 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 19:59:14.287272 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.271915 5935 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 19:59:14.287272 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.271918 5935 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 19:59:14.287272 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.271921 5935 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 19:59:14.287272 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.271923 5935 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 19:59:14.287272 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.271926 5935 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 19:59:14.287272 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.271929 5935 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 19:59:14.287272 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.271931 5935 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 19:59:14.287272 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.271934 5935 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 19:59:14.287272 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.271937 5935 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 19:59:14.287272 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.271940 5935 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 19:59:14.287272 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.271943 5935 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 19:59:14.287272 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.271946 5935 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 19:59:14.287272 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.271948 5935 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 19:59:14.287272 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.271951 5935 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 19:59:14.287272 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.271953 5935 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 19:59:14.287272 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.271956 5935 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 19:59:14.287272 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.271959 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 19:59:14.287272 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.271961 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 19:59:14.287272 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.271964 5935 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 19:59:14.288141 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.271967 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 19:59:14.288141 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.271970 5935 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 19:59:14.288141 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.271972 5935 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 19:59:14.288141 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.271976 5935 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 19:59:14.288141 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.271980 5935 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 19:59:14.288141 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.271984 5935 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 19:59:14.288141 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.271987 5935 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 19:59:14.288141 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.271990 5935 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 19:59:14.288141 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.271993 5935 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 19:59:14.288141 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.271996 5935 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 19:59:14.288141 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.271998 5935 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 19:59:14.288141 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.272001 5935 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 19:59:14.288141 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.272004 5935 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 19:59:14.288141 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.272006 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 19:59:14.288141 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.272009 5935 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 19:59:14.288141 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.272011 5935 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 19:59:14.288141 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.272014 5935 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 19:59:14.288141 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.272017 5935 feature_gate.go:386] unrecognized feature gate: Example Nov 01 19:59:14.288141 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.272019 5935 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 19:59:14.288968 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.272023 5935 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 19:59:14.288968 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.272025 5935 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 19:59:14.288968 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.272028 5935 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 19:59:14.288968 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.272031 5935 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 19:59:14.288968 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.272034 5935 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 19:59:14.288968 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.272036 5935 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 19:59:14.288968 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.272039 5935 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 19:59:14.288968 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.272042 5935 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 19:59:14.288968 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.272044 5935 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 19:59:14.288968 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.272047 5935 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 19:59:14.288968 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.272050 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 19:59:14.288968 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.272052 5935 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 19:59:14.288968 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.272055 5935 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 19:59:14.288968 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.272058 5935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 19:59:14.288968 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.272060 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 19:59:14.288968 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.272063 5935 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 19:59:14.288968 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.272065 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 19:59:14.288968 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.272068 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 19:59:14.288968 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.272071 5935 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 19:59:14.288968 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.272074 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 19:59:14.289892 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.272076 5935 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 19:59:14.289892 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.272079 5935 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 19:59:14.289892 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.272082 5935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 19:59:14.289892 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.272084 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 19:59:14.289892 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.272088 5935 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 19:59:14.289892 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.272091 5935 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 19:59:14.289892 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.272093 5935 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 19:59:14.289892 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.272096 5935 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 19:59:14.289892 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.272098 5935 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 19:59:14.289892 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.272101 5935 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 19:59:14.289892 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.272104 5935 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 19:59:14.289892 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.272106 5935 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 19:59:14.289892 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.272109 5935 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 19:59:14.289892 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.272112 5935 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 19:59:14.289892 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.272114 5935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 19:59:14.289892 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.272117 5935 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 19:59:14.289892 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.272119 5935 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 19:59:14.289892 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.272122 5935 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 19:59:14.289892 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.272124 5935 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 19:59:14.289892 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.272127 5935 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 19:59:14.290834 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.272129 5935 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 19:59:14.290834 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.272132 5935 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 19:59:14.290834 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.272134 5935 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 19:59:14.290834 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.272137 5935 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 19:59:14.290834 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.272140 5935 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 19:59:14.290834 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.272143 5935 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 19:59:14.290834 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.272148 5935 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 19:59:14.290834 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.272152 5935 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 19:59:14.290834 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.272155 5935 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 19:59:14.290834 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.272157 5935 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 19:59:14.290834 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.272160 5935 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 19:59:14.290834 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.272163 5935 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 19:59:14.290834 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.272166 5935 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 19:59:14.290834 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.272168 5935 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 19:59:14.290834 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.272171 5935 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 19:59:14.290834 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.272174 5935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 19:59:14.290834 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.272176 5935 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 19:59:14.290834 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.272179 5935 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 19:59:14.290834 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.272181 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 19:59:14.291585 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.272183 5935 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 19:59:14.291585 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.272186 5935 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 19:59:14.291585 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.272213 5935 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 19:59:14.291585 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273116 5935 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 19:59:14.291585 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273134 5935 feature_gate.go:386] unrecognized feature gate: Example Nov 01 19:59:14.291585 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273140 5935 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 19:59:14.291585 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273144 5935 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 19:59:14.291585 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273149 5935 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 19:59:14.291585 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273154 5935 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 19:59:14.291585 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273164 5935 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 19:59:14.291585 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273168 5935 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 19:59:14.291585 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273172 5935 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 19:59:14.291585 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273177 5935 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 19:59:14.291585 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273182 5935 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 19:59:14.291585 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273186 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 19:59:14.291585 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273190 5935 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 19:59:14.292018 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273194 5935 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 19:59:14.292018 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273198 5935 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 19:59:14.292018 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273203 5935 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 19:59:14.292018 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273207 5935 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 19:59:14.292018 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273212 5935 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 19:59:14.292018 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273221 5935 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 19:59:14.292018 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273225 5935 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 19:59:14.292018 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273229 5935 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 19:59:14.292018 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273234 5935 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 19:59:14.292018 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273238 5935 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 19:59:14.292018 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273242 5935 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 19:59:14.292018 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273246 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 19:59:14.292018 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273251 5935 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 19:59:14.292018 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273255 5935 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 19:59:14.292018 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273260 5935 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 19:59:14.292018 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273264 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 19:59:14.292018 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273268 5935 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 19:59:14.292018 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273273 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 19:59:14.292018 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273281 5935 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 19:59:14.292018 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273304 5935 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 19:59:14.292543 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273309 5935 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 19:59:14.292543 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273314 5935 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 19:59:14.292543 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273318 5935 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 19:59:14.292543 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273323 5935 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 19:59:14.292543 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273328 5935 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 19:59:14.292543 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273332 5935 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 19:59:14.292543 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273336 5935 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 19:59:14.292543 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273343 5935 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 19:59:14.292543 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273417 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 19:59:14.292543 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273424 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 19:59:14.292543 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273431 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 19:59:14.292543 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273435 5935 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 19:59:14.292543 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273441 5935 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 19:59:14.292543 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273462 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 19:59:14.292543 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273477 5935 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 19:59:14.292543 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273480 5935 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 19:59:14.292543 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273572 5935 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 19:59:14.292543 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273577 5935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 19:59:14.292543 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273580 5935 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 19:59:14.292543 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273583 5935 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 19:59:14.293031 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273586 5935 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 19:59:14.293031 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273589 5935 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 19:59:14.293031 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273592 5935 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 19:59:14.293031 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273594 5935 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 19:59:14.293031 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273597 5935 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 19:59:14.293031 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273600 5935 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 19:59:14.293031 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273602 5935 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 19:59:14.293031 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273605 5935 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 19:59:14.293031 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273608 5935 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 19:59:14.293031 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273610 5935 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 19:59:14.293031 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273613 5935 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 19:59:14.293031 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273615 5935 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 19:59:14.293031 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273620 5935 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 19:59:14.293031 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273624 5935 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 19:59:14.293031 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273629 5935 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 19:59:14.293031 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273631 5935 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 19:59:14.293031 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273634 5935 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 19:59:14.293031 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273638 5935 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 19:59:14.293031 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273640 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 19:59:14.293505 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273643 5935 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 19:59:14.293505 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273647 5935 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 19:59:14.293505 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273651 5935 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 19:59:14.293505 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273654 5935 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 19:59:14.293505 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273656 5935 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 19:59:14.293505 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273659 5935 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 19:59:14.293505 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273661 5935 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 19:59:14.293505 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273664 5935 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 19:59:14.293505 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273667 5935 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 19:59:14.293505 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273669 5935 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 19:59:14.293505 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273672 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 19:59:14.293505 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273674 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 19:59:14.293505 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273677 5935 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 19:59:14.293505 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273679 5935 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 19:59:14.293505 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273682 5935 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 19:59:14.293505 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273685 5935 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 19:59:14.293505 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273687 5935 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 19:59:14.293505 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273690 5935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 19:59:14.293505 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273692 5935 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 19:59:14.293978 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273695 5935 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 19:59:14.293978 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273697 5935 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 19:59:14.293978 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273700 5935 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 19:59:14.293978 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273703 5935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 19:59:14.293978 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273705 5935 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 19:59:14.293978 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273708 5935 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 19:59:14.293978 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273711 5935 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 19:59:14.293978 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273713 5935 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 19:59:14.293978 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273716 5935 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 19:59:14.293978 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273719 5935 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 19:59:14.293978 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273722 5935 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 19:59:14.293978 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273724 5935 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 19:59:14.293978 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273727 5935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 19:59:14.293978 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273729 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 19:59:14.293978 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273733 5935 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 19:59:14.294371 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.273769 5935 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 01 19:59:14.294371 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273806 5935 feature_gate.go:386] unrecognized feature gate: Example2 Nov 01 19:59:14.294371 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273811 5935 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 01 19:59:14.294371 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273814 5935 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 01 19:59:14.294371 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273816 5935 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 01 19:59:14.294371 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273820 5935 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 01 19:59:14.294371 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273824 5935 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 01 19:59:14.294371 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273827 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 01 19:59:14.294371 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273831 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 01 19:59:14.294371 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273834 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 01 19:59:14.294371 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273837 5935 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 01 19:59:14.294371 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273840 5935 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 01 19:59:14.294371 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273843 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 01 19:59:14.294371 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273846 5935 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 01 19:59:14.294371 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273849 5935 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 01 19:59:14.294371 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273851 5935 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 01 19:59:14.294799 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273854 5935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 01 19:59:14.294799 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273856 5935 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 01 19:59:14.294799 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273859 5935 feature_gate.go:386] unrecognized feature gate: EVPN Nov 01 19:59:14.294799 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273861 5935 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 01 19:59:14.294799 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273864 5935 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 01 19:59:14.294799 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273866 5935 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 01 19:59:14.294799 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273869 5935 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 01 19:59:14.294799 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273871 5935 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 01 19:59:14.294799 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273874 5935 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 01 19:59:14.294799 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273877 5935 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 01 19:59:14.294799 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273880 5935 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 01 19:59:14.294799 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273882 5935 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 01 19:59:14.294799 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273885 5935 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 01 19:59:14.294799 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273887 5935 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 01 19:59:14.294799 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273890 5935 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 01 19:59:14.294799 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273894 5935 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 01 19:59:14.294799 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273897 5935 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 01 19:59:14.294799 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273900 5935 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 01 19:59:14.294799 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273910 5935 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 01 19:59:14.294799 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273913 5935 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 01 19:59:14.295450 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273916 5935 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 01 19:59:14.295450 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273918 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 01 19:59:14.295450 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273921 5935 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 01 19:59:14.295450 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273924 5935 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 01 19:59:14.295450 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273927 5935 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 01 19:59:14.295450 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273929 5935 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 01 19:59:14.295450 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273932 5935 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 01 19:59:14.295450 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273935 5935 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 01 19:59:14.295450 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273938 5935 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 01 19:59:14.295450 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273941 5935 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 01 19:59:14.295450 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273943 5935 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 01 19:59:14.295450 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273946 5935 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 01 19:59:14.295450 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273948 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 01 19:59:14.295450 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273951 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 01 19:59:14.295450 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273953 5935 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 01 19:59:14.295450 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273956 5935 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 01 19:59:14.295450 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273958 5935 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 01 19:59:14.295450 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273961 5935 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 01 19:59:14.295450 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273963 5935 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 01 19:59:14.296440 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273966 5935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 01 19:59:14.296440 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273968 5935 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 01 19:59:14.296440 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273971 5935 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 01 19:59:14.296440 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273973 5935 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 01 19:59:14.296440 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273976 5935 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 01 19:59:14.296440 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273978 5935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 01 19:59:14.296440 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273981 5935 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 01 19:59:14.296440 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273983 5935 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 01 19:59:14.296440 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273986 5935 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 01 19:59:14.296440 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273988 5935 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 01 19:59:14.296440 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273991 5935 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 01 19:59:14.296440 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273993 5935 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 01 19:59:14.296440 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.273997 5935 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 01 19:59:14.296440 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.274014 5935 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 01 19:59:14.296440 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.274017 5935 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 01 19:59:14.296440 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.274019 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 01 19:59:14.296440 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.274022 5935 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 01 19:59:14.296440 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.274024 5935 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 01 19:59:14.296440 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.274029 5935 feature_gate.go:386] unrecognized feature gate: Example Nov 01 19:59:14.297153 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.274032 5935 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 01 19:59:14.297153 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.274035 5935 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 01 19:59:14.297153 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.274038 5935 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 01 19:59:14.297153 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.274040 5935 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 01 19:59:14.297153 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.274043 5935 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 01 19:59:14.297153 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.274046 5935 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 01 19:59:14.297153 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.274048 5935 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 01 19:59:14.297153 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.274051 5935 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 01 19:59:14.297153 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.274054 5935 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 01 19:59:14.297153 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.274056 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 01 19:59:14.297153 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.274059 5935 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 01 19:59:14.297153 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.274061 5935 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 01 19:59:14.297153 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.274064 5935 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 01 19:59:14.297153 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.274067 5935 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 01 19:59:14.297153 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.274069 5935 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 01 19:59:14.297153 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.274072 5935 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 01 19:59:14.297153 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.274074 5935 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 01 19:59:14.297153 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.274077 5935 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 01 19:59:14.297153 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.274080 5935 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 01 19:59:14.297153 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.274082 5935 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 01 19:59:14.297678 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.274085 5935 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 01 19:59:14.297678 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.274087 5935 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 01 19:59:14.297678 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.274090 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 01 19:59:14.297678 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.274092 5935 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 01 19:59:14.297678 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.274095 5935 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 01 19:59:14.297678 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.274098 5935 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 01 19:59:14.297678 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.274100 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 01 19:59:14.297678 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.274103 5935 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 01 19:59:14.297678 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.274106 5935 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 01 19:59:14.297678 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.274108 5935 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 01 19:59:14.297678 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.274111 5935 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 01 19:59:14.297678 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.274115 5935 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 01 19:59:14.297678 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:14.274119 5935 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 01 19:59:14.297678 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.274171 5935 server.go:957] "Client rotation is on, will bootstrap in background" Nov 01 19:59:14.297678 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.275122 5935 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 01 19:59:14.297678 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.275177 5935 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 01 19:59:14.297678 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.275366 5935 server.go:1014] "Starting client certificate rotation" Nov 01 19:59:14.297678 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.275498 5935 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 01 19:59:14.298220 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.275542 5935 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 19:53:02 +0000 UTC" deadline="2028-04-22 08:43:50.752867384 +0000 UTC" Nov 01 19:59:14.298220 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.275568 5935 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12900h44m36.477300934s" Nov 01 19:59:14.298220 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.275832 5935 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 19:59:14.298220 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.276859 5935 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 01 19:59:14.298220 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.277978 5935 log.go:25] "Validated CRI v1 runtime API" Nov 01 19:59:14.298220 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.279799 5935 log.go:25] "Validated CRI v1 image API" Nov 01 19:59:14.298220 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.280529 5935 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 01 19:59:14.298220 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.283681 5935 fs.go:135] Filesystem UUIDs: map[28472917-cd5a-4041-9608-8bfe395becb0:/dev/nvme0n1p3 5ca2b1ae-7a09-4187-9f92-788c4705a8fd:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Nov 01 19:59:14.298472 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.283707 5935 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/09870c8784bd520f17a437877c0c526f28d0f3380a6e8f5aed986a69fec213db/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/09870c8784bd520f17a437877c0c526f28d0f3380a6e8f5aed986a69fec213db/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0b5b365de0b6be1ee91aec65b04995e35d9cccda934e4cb6491bcde0f1bcf0b0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0b5b365de0b6be1ee91aec65b04995e35d9cccda934e4cb6491bcde0f1bcf0b0/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/171709489a943ced99b7c4119f166be2a9704fe76d0e017f1d427be14614f522/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/171709489a943ced99b7c4119f166be2a9704fe76d0e017f1d427be14614f522/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1eb923680684f91bc5bb79f36fc319787360453b2d2fdf254d2ed40e1b919e45/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1eb923680684f91bc5bb79f36fc319787360453b2d2fdf254d2ed40e1b919e45/userdata/shm major:0 minor:361 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/21eeb2fddeba4c6a0109132af0db2fdc6635c8a149fcaa1750e43d852575ed2f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/21eeb2fddeba4c6a0109132af0db2fdc6635c8a149fcaa1750e43d852575ed2f/userdata/shm major:0 minor:352 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/23796b1cbe6433b0a4bdabc415c684b57ffbafc67eda1d4bbfd8bb577981ba61/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/23796b1cbe6433b0a4bdabc415c684b57ffbafc67eda1d4bbfd8bb577981ba61/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3b6395bcdb9ea079988584faf913c7ac86c2a5cb495fcfd707a412590ea159ad/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3b6395bcdb9ea079988584faf913c7ac86c2a5cb495fcfd707a412590ea159ad/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/45dad8cb2d5f132939bc7e5aec60c3e23396ae5e309ff9f31a12cd963cbf8a97/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/45dad8cb2d5f132939bc7e5aec60c3e23396ae5e309ff9f31a12cd963cbf8a97/userdata/shm major:0 minor:231 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4fa5c3abb6e944f8ba628a5f163d42f5e86eb83f848be3bb992e2bca3e6d42c0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4fa5c3abb6e944f8ba628a5f163d42f5e86eb83f848be3bb992e2bca3e6d42c0/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/56edd4df9c723fdf0a862ebdda0d5030f1d5fab4ebdd28d11ef94b80452b733c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/56edd4df9c723fdf0a862ebdda0d5030f1d5fab4ebdd28d11ef94b80452b733c/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/66ab60379d96b63f8e157420e9e45dd63f147c34d12ebee2405bbee10566620f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/66ab60379d96b63f8e157420e9e45dd63f147c34d12ebee2405bbee10566620f/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/792959e7e29e92310ac6d7f92cafef9566d8b05c6fa9973efacad40daefd3c11/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/792959e7e29e92310ac6d7f92cafef9566d8b05c6fa9973efacad40daefd3c11/userdata/shm major:0 minor:295 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/849c0f8d3b237ee06cab1e3c239405786fcf922f40061978e92bdd14c3ce4172/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/849c0f8d3b237ee06cab1e3c239405786fcf922f40061978e92bdd14c3ce4172/userdata/shm major:0 minor:359 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9553a008d6925c67e4a2b603ebe7471784b7851172e5102a19c033372f1fad59/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9553a008d6925c67e4a2b603ebe7471784b7851172e5102a19c033372f1fad59/userdata/shm major:0 minor:363 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9a17db04c7c5522bcea5dd87ba07081512f9792373234ba984f754b9334588ff/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9a17db04c7c5522bcea5dd87ba07081512f9792373234ba984f754b9334588ff/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a995ac4070f05fefebeaa4cf8120b1fba5b3082daff99e951f84d285c1583ead/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a995ac4070f05fefebeaa4cf8120b1fba5b3082daff99e951f84d285c1583ead/userdata/shm major:0 minor:278 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/afee1e107bc314024deaf12c05f81a70515de378ec416a05bc0a75c3eb5c478d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/afee1e107bc314024deaf12c05f81a70515de378ec416a05bc0a75c3eb5c478d/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b3ea44655cb46e9c5ef929ad76a73cb9490badcb9c5405bde8f151cfa4aeb8c8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b3ea44655cb46e9c5ef929ad76a73cb9490badcb9c5405bde8f151cfa4aeb8c8/userdata/shm major:0 minor:382 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e7771648236c3606589fa0b023edcfea1aac1c04baba12589d64b147cd4f6be6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e7771648236c3606589fa0b023edcfea1aac1c04baba12589d64b147cd4f6be6/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e943b81be985021134f7b01fa18e9de2d046185a015ab296c031b2318a819812/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e943b81be985021134f7b01fa18e9de2d046185a015ab296c031b2318a819812/userdata/shm major:0 minor:254 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ebf7ce221b34e42276e46bea7df404696c67456b71aa0f2cc734056492424635/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ebf7ce221b34e42276e46bea7df404696c67456b71aa0f2cc734056492424635/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fb54b0679fdeb76219efab65f1a90e7ad44ae01bad5a0a5df84fc00117c83331/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fb54b0679fdeb76219efab65f1a90e7ad44ae01bad5a0a5df84fc00117c83331/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/16e51d04-90d5-484d-9f85-e9d484205220/volumes/kubernetes.io~projected/kube-api-access-vq4k2:{mountpoint:/var/lib/kubelet/pods/16e51d04-90d5-484d-9f85-e9d484205220/volumes/kubernetes.io~projected/kube-api-access-vq4k2 major:0 minor:358 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1cd0e9b2-272f-46f0-88d3-89f58c5dab49/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/1cd0e9b2-272f-46f0-88d3-89f58c5dab49/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1cd0e9b2-272f-46f0-88d3-89f58c5dab49/volumes/kubernetes.io~projected/kube-api-access-6dg6b:{mountpoint:/var/lib/kubelet/pods/1cd0e9b2-272f-46f0-88d3-89f58c5dab49/volumes/kubernetes.io~projected/kube-api-access-6dg6b major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1cd0e9b2-272f-46f0-88d3-89f58c5dab49/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/1cd0e9b2-272f-46f0-88d3-89f58c5dab49/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2efc5bbf-efd4-4a42-bed8-53f2490bb7bc/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/2efc5bbf-efd4-4a42-bed8-53f2490bb7bc/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:357 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40d57e7e-4896-4e0c-b266-ef5e6ca8beab/volumes/kubernetes.io~projected/kube-api-access-nq6fc:{mountpoint:/var/lib/kubelet/pods/40d57e7e-4896-4e0c-b266-ef5e6ca8beab/volumes/kubernetes.io~projected/kube-api-access-nq6fc major:0 minor:294 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40d57e7e-4896-4e0c-b266-ef5e6ca8beab/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/40d57e7e-4896-4e0c-b266-ef5e6ca8beab/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40d57e7e-4896-4e0c-b266-ef5e6ca8beab/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/40d57e7e-4896-4e0c-b266-ef5e6ca8beab/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40d57e7e-4896-4e0c-b266-ef5e6ca8beab/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/40d57e7e-4896-4e0c-b266-ef5e6ca8beab/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40d57e7e-4896-4e0c-b266-ef5e6ca8beab/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/40d57e7e-4896-4e0c-b266-ef5e6ca8beab/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:292 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40d57e7e-4896-4e0c-b266-ef5e6ca8beab/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/40d57e7e-4896-4e0c-b266-ef5e6ca8beab/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40d57e7e-4896-4e0c-b266-ef5e6ca8beab/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/40d57e7e-4896-4e0c-b266-ef5e6ca8beab/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/58167352-076f-40d7-abee-1c5f48f67566/volumes/kubernetes.io~projected/kube-api-access-chtj4:{mountpoint:/var/lib/kubelet/pods/58167352-076f-40d7-abee-1c5f48f67566/volumes/kubernetes.io~projected/kube-api-access-chtj4 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/58167352-076f-40d7-abee-1c5f48f67566/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/58167352-076f-40d7-abee-1c5f48f67566/volumes/kubernetes.io~secret/metrics-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a57adad-0c11-4902-be16-b8dfb4e3ee13/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/5a57adad-0c11-4902-be16-b8dfb4e3ee13/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a57adad-0c11-4902-be16-b8dfb4e3ee13/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/5a57adad-0c11-4902-be16-b8dfb4e3ee13/volumes/kubernetes.io~empty-dir/tmp major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a57adad-0c11-4902-be16-b8dfb4e3ee13/volumes/kubernetes.io~projected/kube-api-access-98mhk:{mountpoint:/var/lib/kubelet/pods/5a57adad-0c11-4902-be16-b8dfb4e3ee13/volumes/kubernetes.io~projected/kube-api-access-98mhk major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/60e3a171-27e7-4ae6-ad6c-89984ecf5634/volumes/kubernetes.io~projected/kube-api-access-4cq6h:{mountpoint:/var/lib/kubelet/pods/60e3a171-27e7-4ae6-ad6c-89984ecf5634/volumes/kubernetes.io~projected/kube-api-access-4cq6h major:0 minor:375 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/60e3a171-27e7-4ae6-ad6c-89984ecf5634/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/60e3a171-27e7-4ae6-ad6c-89984ecf5634/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:373 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/60e3a171-27e7-4ae6-ad6c-89984ecf5634/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/60e3a171-27e7-4ae6-ad6c-89984ecf5634/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:374 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/62c9da55-b21d-4f8f-a3ec-98e823655351/volumes/kubernetes.io~projected/kube-api-access-fhv76:{mountpoint:/var/lib/kubelet/pods/62c9da55-b21d-4f8f-a3ec-98e823655351/volumes/kubernetes.io~projected/kube-api-access-fhv76 major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/65e6a200-c6f4-4671-9d2e-b0158ff3c5ce/volumes/kubernetes.io~projected/kube-api-access-jtnhd:{mountpoint:/var/lib/kubelet/pods/65e6a200-c6f4-4671-9d2e-b0158ff3c5ce/volumes/kubernetes.io~projected/kube-api-access-jtnhd major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/65e6a200-c6f4-4671-9d2e-b0158ff3c5ce/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/65e6a200-c6f4-4671-9d2e-b0158ff3c5ce/volumes/kubernetes.io~secret/metrics-certs major:0 minor:356 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/80c66ec7-5430-41a1-8641-56db5e4e9d73/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/80c66ec7-5430-41a1-8641-56db5e4e9d73/volume-subpaths/web-config/config-reloader/2 major:0 minor:342 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/80c66ec7-5430-41a1-8641-56db5e4e9d73/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/80c66ec7-5430-41a1-8641-56db5e4e9d73/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:342 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/80c66ec7-5430-41a1-8641-56db5e4e9d73/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/80c66ec7-5430-41a1-8641-56db5e4e9d73/volume-subpaths/web-config/prometheus/15 major:0 minor:342 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/80c66ec7-5430-41a1-8641-56db5e4e9d73/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/80c66ec7-5430-41a1-8641-56db5e4e9d73/volumes/kubernetes.io~empty-dir/config-out major:0 minor:345 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/80c66ec7-5430-41a1-8641-56db5e4e9d73/volumes/kubernetes.io~projected/kube-api-access-x4zhc:{mountpoint:/var/lib/kubelet/pods/80c66ec7-5430-41a1-8641-56db5e4e9d73/volumes/kubernetes.io~projected/kube-api-access-x4zhc major:0 minor:351 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/80c66ec7-5430-41a1-8641-56db5e4e9d73/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/80c66ec7-5430-41a1-8641-56db5e4e9d73/volumes/kubernetes.io~projected/tls-assets major:0 minor:346 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/80c66ec7-5430-41a1-8641-56db5e4e9d73/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/80c66ec7-5430-41a1-8641-56db5e4e9d73/volumes/kubernetes.io~secret/config major:0 minor:349 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/80c66ec7-5430-41a1-8641-56db5e4e9d73/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/80c66ec7-5430-41a1-8641-56db5e4e9d73/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:344 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/80c66ec7-5430-41a1-8641-56db5e4e9d73/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/80c66ec7-5430-41a1-8641-56db5e4e9d73/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:347 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/80c66ec7-5430-41a1-8641-56db5e4e9d73/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/80c66ec7-5430-41a1-8641-56db5e4e9d73/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:348 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/80c66ec7-5430-41a1-8641-56db5e4e9d73/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/80c66ec7-5430-41a1-8641-56db5e4e9d73/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:343 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/80c66ec7-5430-41a1-8641-56db5e4e9d73/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/80c66ec7-5430-41a1-8641-56db5e4e9d73/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:341 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/80c66ec7-5430-41a1-8641-56db5e4e9d73/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/80c66ec7-5430-41a1-8641-56db5e4e9d73/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:350 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/80c66ec7-5430-41a1-8641-56db5e4e9d73/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/80c66ec7-5430-41a1-8641-56db5e4e9d73/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:332 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/80c66ec7-5430-41a1-8641-56db5e4e9d73/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/80c66ec7-5430-41a1-8641-56db5e4e9d73/volumes/kubernetes.io~secret/web-config major:0 minor:342 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/83a0642b-4dfe-4ddb-b506-f0654c7b6d30/volumes/kubernetes.io~projected/kube-api-access-bm8bk:{mountpoint:/var/lib/kubelet/pods/83a0642b-4dfe-4ddb-b506-f0654c7b6d30/volumes/kubernetes.io~projected/kube-api-access-bm8bk major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/83a0642b-4dfe-4ddb-b506-f0654c7b6d30/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/83a0642b-4dfe-4ddb-b506-f0654c7b6d30/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8537e78c-5b2c-4966-9558-044ddc39119b/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/8537e78c-5b2c-4966-9558-044ddc39119b/volumes/kubernetes.io~secret/agent-certs major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8/volumes/kubernetes.io~projected/kube-api-access-pcwgs:{mountpoint:/var/lib/kubelet/pods/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8/volumes/kubernetes.io~projected/kube-api-access-pcwgs major:0 minor:253 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:248 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:252 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/91ef232e-99ae-42f2-be58-e17cc42f12ea/volumes/kubernetes.io~projected/kube-api-access-8z6z7:{mountpoint:/var/lib/kubelet/pods/91ef232e-99ae-42f2-be58-e17cc42f12ea/volumes/kubernetes.io~projected/kube-api-access-8z6z7 major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/961c6886-e297-4202-b10a-df97f3537736/volumes/kubernetes.io~projected/kube-api-access-x5bdv:{mountpoint:/var/lib/kubelet/pods/961c6886-e297-4202-b10a-df97f3537736/volumes/kubernetes.io~projected/kube-api-access-x5bdv major:0 minor:277 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/961c6886-e297-4202-b10a-df97f3537736/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/961c6886-e297-4202-b10a-df97f3537736/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:272 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/961c6886-e297-4202-b10a-df97f3537736/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/961c6886-e297-4202-b10a-df97f3537736/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:266 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/983a921d-61a5-4668-8e17-1a24b6eaf8b6/volumes/kubernetes.io~projected/kube-api-access-8fntm:{mountpoint:/var/lib/kubelet/pods/983a921d-61a5-4668-8e17-1a24b6eaf8b6/volumes/kubernetes.io~projected/kube-api-access-8fntm major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e564268-fa85-48e6-8d1c-5069cc0f9648/volumes/kubernetes.io~projected/kube-api-access-mnjs4:{mountpoint:/var/lib/kubelet/pods/9e564268-fa85-48e6-8d1c-5069cc0f9648/volumes/kubernetes.io~projected/kube-api-access-mnjs4 major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a225113f-cd49-48a9-b211-595bf5de763a/volumes/kubernetes.io~projected/kube-api-access-fs2sl:{mountpoint:/var/lib/kubelet/pods/a225113f-cd49-48a9-b211-595bf5de763a/volumes/kubernetes.io~projected/kube-api-access-fs2sl major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a225113f-cd49-48a9-b211-595bf5de763a/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/a225113f-cd49-48a9-b211-595bf5de763a/volumes/kubernetes.io~secret/cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c0e3a247-a792-431d-85cb-9b46982c3373/volumes/kubernetes.io~projected/kube-api-access-nz4mx:{mountpoint:/var/lib/kubelet/pods/c0e3a247-a792-431d-85cb-9b46982c3373/volumes/kubernetes.io~projected/kube-api-access-nz4mx major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f16f57a8-9630-416c-bd95-f5a2c6af2f15/volumes/kubernetes.io~projected/kube-api-access-j9xlj:{mountpoint:/var/lib/kubelet/pods/f16f57a8-9630-416c-bd95-f5a2c6af2f15/volumes/kubernetes.io~projected/kube-api-access-j9xlj major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fa82c276-5679-40f4-b7b4-e73eba7de9ef/volumes/kubernetes.io~projected/kube-api-access-g7jcx:{mountpoint:/var/lib/kubelet/pods/fa82c276-5679-40f4-b7b4-e73eba7de9ef/volumes/kubernetes.io~projected/kube-api-access-g7jcx major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fa82c276-5679-40f4-b7b4-e73eba7de9ef/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/fa82c276-5679-40f4-b7b4-e73eba7de9ef/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fa82c276-5679-40f4-b7b4-e73eba7de9ef/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/fa82c276-5679-40f4-b7b4-e73eba7de9ef/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:229 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/d4edadbb9c41536eb061faf6b50d71cf366cc56cb5411fb3d57cae048c137931/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/fe5373c95aca65f9c5e6c4457e7b240873b8263265815890fdffa6a9f6457c2d/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/8fd911b50bfb1d792d1f9df9293081d41b637506575e973941537c8a15cf22d3/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/c12ec0f819215117d55a6b9582b3f5aa572035260f55a391ebc34ce7c851b451/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/93c911d38b381a9f5adfdb47640bc0fd4f7dd1a7b9fb653f2455c1fd2085620c/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/c064ea10851117c45c7819214c892ae20eabc207a1f98333d1c5e65388cc371b/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/22f0730700a7d08cddee7f2b97d59c3b2b7efa0055ffa925e7648ef7ce58d732/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/e4ba904c343dcbc3dfc066aac90f650b2139194152ffcb4252c01a59281b2fcd/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/e37dfe7cbc9db5b22e8dd520c37a99aa96bd6dace11fa2e19e5fb975e10fc13d/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/a26c8d62ecc8d09ff2a1a166b31731d1ce895848f4282ff757d12530f8b23001/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/5955f954da3c6ac5e61527ae243d6128145500da98c5893ed590242ef4aeabaf/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/3a05195a2993b6422c7a57adf19f4a2099ed521980756079ed2dc3085b44998a/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/af408293995673fedd255cf4cdc67d54a9a6b7dafa78806d83da550ddf0f89f4/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/2e075ca20ec466b70b5fbc66bb3fafd7ec2622a96312423b018d1b7e3b0d4dc3/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/d7e21d3429589ada079d50af922643bcca8c8a70f52b9217a8f0e166ff0b20bd/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/c7568a0867fe250fd8e9fa0d6ba9c233ed9e3b8f0769c0bfb299f0ad92db1544/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/73a3895c1f619dab2d8f1b9cba0787cd15b441dd102754f94ac0fa780a3516bc/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/294949e263446645bce745101e3b33226b234204e9a179d642212b6301f2f253/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/37cfc1705a0fc402a81d872634389e230f95b2b4ad77b3ab1c508b055d7aa63a/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/bc3a8d70cbdd8a2d58d934832beaf93554d602cc6095ede5bdb9a455b44c3851/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/0dffa1eba67c3041cefe6a540de8d8c2a9a531fb4d322d2346dbf84b3dc6c9d1/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/da8d1b93bf6ae6cd95566070f5cc0d0e00a42624df888c1e7492f5120f7f0857/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/cce7a863937ade38d6553f113bac67ad28ff23caa5823338469ad719db810668/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/de48ff5dd6a253b9d467dd9ac3a9a836735aa23394ed900230d7a33ab7ce56f0/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/202a7a53cae1b56b0d2bfc7d1f827dc77827f12e2a032dbed872ed8c1ba4f182/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/334f57b97396068d85daf574ec4de1260d5fce1edaa6311af91da420f70c7a10/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/6813ba35fafc4b5621a6a5561967bcb3f21cf8e326193b388184e1165c170963/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/baaec2b4a39313a23eb8dbc550057255566de56273afbb5b8d5ccedd6c70caf9/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/1fed80843a48e48b452024044d59b2962163949fb05db424ca586994717a9f8f/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-243:{mountpoint:/var/lib/containers/storage/overlay/c7ef59c57db45caae7b73b537316d0ffa07e6973609128fee9ec9b669bf0996a/merged major:0 minor:243 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/7fc4b3da1156cec3a14676db61fe23380286b5134cfffb521108d1bb30130702/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/f5f494d9965f9b438d8ff9458588c90054280e672a67a2f8a9bdd7ac1e1f59dc/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/68d158ff62c04034617749c67e444ae22ad8bb43a8c17eccebb7a0183a9d79bf/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/8e90504474ae6d4c76833dc54aac6fe8cd8254b45463f5af1394386be723bfd2/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/559f90160852ac482fdd82016733ada978653a134460b9b0bf09c11874870904/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-282:{mountpoint:/var/lib/containers/storage/overlay/408863f1ca1c5ddbc9032bc5e5380eb7b09d4e41a59f9efc8ec1ac52f33b55c9/merged major:0 minor:282 fsType:overlay blockSize:0} overlay_0-297:{mountpoint:/var/lib/containers/storage/overlay/69ec18f1a23cfa55b43e2f6b4b09531b4262540be5fdf1d31c159b55eab69584/merged major:0 minor:297 fsType:overlay blockSize:0} overlay_0-299:{mountpoint:/var/lib/containers/storage/overlay/84f023f3c86c726394e7261d4f6306bbf429adfed0e8daf909f1d459b9d0015e/merged major:0 minor:299 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/6c3621d1a0c2e2cbe8079b7d52fdf19cd2562d8926cfb9a62a8a44d4a88103ec/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-309:{mountpoint:/var/lib/containers/storage/overlay/177eefea2e12ad3898a2530b784161acc62982f87a662afb19c98f2df5862291/merged major:0 minor:309 fsType:overlay blockSize:0} overlay_0-311:{mountpoint:/var/lib/containers/storage/overlay/4d670a24f3f1944b5a7a3c446764f320c2c98c1a240a1d5946ffc9b55f1331af/merged major:0 minor:311 fsType:overlay blockSize:0} overlay_0-317:{mountpoint:/var/lib/containers/storage/overlay/9a43a81cda7fc3319cba809ddedabe87d6f991b5fc640df4e93a38de992a314a/merged major:0 minor:317 fsType:overlay blockSize:0} overlay_0-322:{mountpoint:/var/lib/containers/storage/overlay/7e6ac13e09c3549e59e02561efccf64e9693058ad60d548d06dbacd33167caff/merged major:0 minor:322 fsType:overlay blockSize:0} overlay_0-327:{mountpoint:/var/lib/containers/storage/overlay/e19c4eb0651ec7014a2675747c244652d00c13017e978b56e8202cfe280793bb/merged major:0 minor:327 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/205a94444a0ec58ba686abde8f55ce4b29ae0fd5ebf1bfe6711936d1322c7b1a/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/d043364bfa902b3793ea7e6155fdfae330745963156ca1953e43df37466910c9/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-365:{mountpoint:/var/lib/containers/storage/overlay/81f8802f769458b4da781cd5c7acdf306450efd4c7b9853e596321c05583b4d6/merged major:0 minor:365 fsType:overlay blockSize:0} overlay_0-367:{mountpoint:/var/lib/containers/storage/overlay/aacc4b88ec8ba21230bd00cb771ac92c09a334e029ee231d93fc94b63a9bc0a8/merged major:0 minor:367 fsType:overlay blockSize:0} overlay_0-369:{mountpoint:/var/lib/containers/storage/overlay/71689ff79f2565de697d366aeb6e60c2bd1e8d272b6b717f488cc70c06b31c48/merged major:0 minor:369 fsType:overlay blockSize:0} overlay_0-371:{mountpoint:/var/lib/containers/storage/overlay/ba25844e012accae26fec8c0bac5a2f9793abd845d9ddc2846fb39db01720eab/merged major:0 minor:371 fsType:overlay blockSize:0} overlay_0-376:{mountpoint:/var/lib/containers/storage/overlay/5355f16b45508f0c06d40967a4e354d1cf7a61fa36801476c22d0238e1ad3dc2/merged major:0 minor:376 fsType:overlay blockSize:0} overlay_0-384:{mountpoint:/var/lib/containers/storage/overlay/37a13f1ee8023714f5940c82f2e5330b01ecead68c9321c729396c434227489a/merged major:0 minor:384 fsType:overlay blockSize:0} overlay_0-389:{mountpoint:/var/lib/containers/storage/overlay/931b442f77c1aad6a950467feec6a1daabe7edbb3010158559aa8e135eb73763/merged major:0 minor:389 fsType:overlay blockSize:0} overlay_0-391:{mountpoint:/var/lib/containers/storage/overlay/b96aa1f959cab7875ed93ccadb15cf6f771580ef31596e8f937b91c25ec874ea/merged major:0 minor:391 fsType:overlay blockSize:0} overlay_0-397:{mountpoint:/var/lib/containers/storage/overlay/359f255a5c665d9eadb3b1223ba71e7264fdac0f642c11be0d69c6f7def67711/merged major:0 minor:397 fsType:overlay blockSize:0} overlay_0-403:{mountpoint:/var/lib/containers/storage/overlay/fa8e95fa3752dd055c821270455efbcada2bb19140628353554dda193b3a26f8/merged major:0 minor:403 fsType:overlay blockSize:0} overlay_0-408:{mountpoint:/var/lib/containers/storage/overlay/8757e3a763459e2b1f4f7c4526f5897ee65a7a85b8ed388f52d3fcae1b7359a1/merged major:0 minor:408 fsType:overlay blockSize:0} overlay_0-410:{mountpoint:/var/lib/containers/storage/overlay/bc85c24efb9aab0e818eed5a8ca7d02427a7f07d42a385c581ca825c47ba98d8/merged major:0 minor:410 fsType:overlay blockSize:0} overlay_0-419:{mountpoint:/var/lib/containers/storage/overlay/396a51f977e5446d9cd71141dff59d4f2d27541e72a497795feee03f4a0741d1/merged major:0 minor:419 fsType:overlay blockSize:0} overlay_0-424:{mountpoint:/var/lib/containers/storage/overlay/2b5cd50a25ef1cab95ed09c80dd3bc7469d42ab44f3eaf1ce8d02994d57b2604/merged major:0 minor:424 fsType:overlay blockSize:0} overlay_0-429:{mountpoint:/var/lib/containers/storage/overlay/10f044fa070de166db933b59932ea3ad4d48e67399f1e656466f827210459e0c/merged major:0 minor:429 fsType:overlay blockSize:0} overlay_0-64:{mountpoint:/var/lib/containers/storage/overlay/e66c420221a35acd3400843e5e4514d15263cc4f52022b857f2f4adbd6370f0c/merged major:0 minor:64 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/d49f98b5d0f92b8486cbf3675199d49c4156df09e61a6e92b28b24dc57042583/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/f08d320b1130531c22c82ca83da24ee99fb37fec32e232470ed24ae374bf4ba3/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/c68ba961848dcaa29a25774ed9bdbd4f91278690011c3c418536fb953c7e16b9/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/1616fe07f5906f05ddb8eb6e16814d856f4d65e7a2a59b01ac6d7fd6818841d5/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/0fab90c2dbde6ecc24ff93fecb897861e87e03677f5746b97394ace3bb7ec0ba/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/6f125ce9819dec79de4051270cdb088b1322096f5bd7b3b73cce9c946dee94ba/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/f5d1591690ed72ad5e0cf626be552ea5f4f718a4040a6807e0c711f4990530b9/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/e3c4c2eb85657f30c76053ebb5605b63c12de60576c6fe53ac55f78c42db7c6c/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/0d523b43e90e252ccee979b14addcdf3ef18182cd64d97996f8f3da80c4f4152/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 01 19:59:14.298767 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.294514 5935 manager.go:217] Machine: {Timestamp:2026-11-01 19:59:14.294026333 +0000 UTC m=+0.067186840 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100232 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2059bf6e77f9d142f81e6481bc6c56 SystemUUID:ec2059bf-6e77-f9d1-42f8-1e6481bc6c56 BootID:47a34720-ea42-4c15-a637-08f630e1e3eb Filesystems:[{Device:/var/lib/kubelet/pods/5a57adad-0c11-4902-be16-b8dfb4e3ee13/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:50 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/40d57e7e-4896-4e0c-b266-ef5e6ca8beab/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:293 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/80c66ec7-5430-41a1-8641-56db5e4e9d73/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:346 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/16e51d04-90d5-484d-9f85-e9d484205220/volumes/kubernetes.io~projected/kube-api-access-vq4k2 DeviceMajor:0 DeviceMinor:358 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:248 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/1cd0e9b2-272f-46f0-88d3-89f58c5dab49/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:48 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/58167352-076f-40d7-abee-1c5f48f67566/volumes/kubernetes.io~projected/kube-api-access-chtj4 DeviceMajor:0 DeviceMinor:202 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/1cd0e9b2-272f-46f0-88d3-89f58c5dab49/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/58167352-076f-40d7-abee-1c5f48f67566/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/83a0642b-4dfe-4ddb-b506-f0654c7b6d30/volumes/kubernetes.io~projected/kube-api-access-bm8bk DeviceMajor:0 DeviceMinor:204 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/80c66ec7-5430-41a1-8641-56db5e4e9d73/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:347 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/80c66ec7-5430-41a1-8641-56db5e4e9d73/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:350 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-429 DeviceMajor:0 DeviceMinor:429 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/961c6886-e297-4202-b10a-df97f3537736/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:266 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/961c6886-e297-4202-b10a-df97f3537736/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:272 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-424 DeviceMajor:0 DeviceMinor:424 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4fa5c3abb6e944f8ba628a5f163d42f5e86eb83f848be3bb992e2bca3e6d42c0/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/56edd4df9c723fdf0a862ebdda0d5030f1d5fab4ebdd28d11ef94b80452b733c/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/40d57e7e-4896-4e0c-b266-ef5e6ca8beab/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:290 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/60e3a171-27e7-4ae6-ad6c-89984ecf5634/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:374 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-309 DeviceMajor:0 DeviceMinor:309 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c0e3a247-a792-431d-85cb-9b46982c3373/volumes/kubernetes.io~projected/kube-api-access-nz4mx DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/5a57adad-0c11-4902-be16-b8dfb4e3ee13/volumes/kubernetes.io~projected/kube-api-access-98mhk DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0b5b365de0b6be1ee91aec65b04995e35d9cccda934e4cb6491bcde0f1bcf0b0/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-384 DeviceMajor:0 DeviceMinor:384 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/83a0642b-4dfe-4ddb-b506-f0654c7b6d30/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-408 DeviceMajor:0 DeviceMinor:408 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/80c66ec7-5430-41a1-8641-56db5e4e9d73/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:342 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-397 DeviceMajor:0 DeviceMinor:397 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1cd0e9b2-272f-46f0-88d3-89f58c5dab49/volumes/kubernetes.io~projected/kube-api-access-6dg6b DeviceMajor:0 DeviceMinor:54 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/66ab60379d96b63f8e157420e9e45dd63f147c34d12ebee2405bbee10566620f/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/45dad8cb2d5f132939bc7e5aec60c3e23396ae5e309ff9f31a12cd963cbf8a97/userdata/shm DeviceMajor:0 DeviceMinor:231 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/80c66ec7-5430-41a1-8641-56db5e4e9d73/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:343 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/80c66ec7-5430-41a1-8641-56db5e4e9d73/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:348 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/171709489a943ced99b7c4119f166be2a9704fe76d0e017f1d427be14614f522/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:252 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/40d57e7e-4896-4e0c-b266-ef5e6ca8beab/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:284 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-282 DeviceMajor:0 DeviceMinor:282 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-389 DeviceMajor:0 DeviceMinor:389 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-297 DeviceMajor:0 DeviceMinor:297 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fb54b0679fdeb76219efab65f1a90e7ad44ae01bad5a0a5df84fc00117c83331/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-311 DeviceMajor:0 DeviceMinor:311 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/60e3a171-27e7-4ae6-ad6c-89984ecf5634/volumes/kubernetes.io~projected/kube-api-access-4cq6h DeviceMajor:0 DeviceMinor:375 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/80c66ec7-5430-41a1-8641-56db5e4e9d73/volumes/kubernetes.io~projected/kube-api-access-x4zhc DeviceMajor:0 DeviceMinor:351 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/65e6a200-c6f4-4671-9d2e-b0158ff3c5ce/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:356 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/23796b1cbe6433b0a4bdabc415c684b57ffbafc67eda1d4bbfd8bb577981ba61/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1eb923680684f91bc5bb79f36fc319787360453b2d2fdf254d2ed40e1b919e45/userdata/shm DeviceMajor:0 DeviceMinor:361 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/60e3a171-27e7-4ae6-ad6c-89984ecf5634/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:373 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9a17db04c7c5522bcea5dd87ba07081512f9792373234ba984f754b9334588ff/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-322 DeviceMajor:0 DeviceMinor:322 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/80c66ec7-5430-41a1-8641-56db5e4e9d73/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:332 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b3ea44655cb46e9c5ef929ad76a73cb9490badcb9c5405bde8f151cfa4aeb8c8/userdata/shm DeviceMajor:0 DeviceMinor:382 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-419 DeviceMajor:0 DeviceMinor:419 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/9e564268-fa85-48e6-8d1c-5069cc0f9648/volumes/kubernetes.io~projected/kube-api-access-mnjs4 DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/80c66ec7-5430-41a1-8641-56db5e4e9d73/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:345 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/09870c8784bd520f17a437877c0c526f28d0f3380a6e8f5aed986a69fec213db/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e7771648236c3606589fa0b023edcfea1aac1c04baba12589d64b147cd4f6be6/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-299 DeviceMajor:0 DeviceMinor:299 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/afee1e107bc314024deaf12c05f81a70515de378ec416a05bc0a75c3eb5c478d/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/62c9da55-b21d-4f8f-a3ec-98e823655351/volumes/kubernetes.io~projected/kube-api-access-fhv76 DeviceMajor:0 DeviceMinor:53 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/80c66ec7-5430-41a1-8641-56db5e4e9d73/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:349 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/983a921d-61a5-4668-8e17-1a24b6eaf8b6/volumes/kubernetes.io~projected/kube-api-access-8fntm DeviceMajor:0 DeviceMinor:52 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ebf7ce221b34e42276e46bea7df404696c67456b71aa0f2cc734056492424635/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-243 DeviceMajor:0 DeviceMinor:243 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2efc5bbf-efd4-4a42-bed8-53f2490bb7bc/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:357 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/40d57e7e-4896-4e0c-b266-ef5e6ca8beab/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:289 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/80c66ec7-5430-41a1-8641-56db5e4e9d73/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:344 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/40d57e7e-4896-4e0c-b266-ef5e6ca8beab/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:292 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-317 DeviceMajor:0 DeviceMinor:317 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8537e78c-5b2c-4966-9558-044ddc39119b/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:47 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a995ac4070f05fefebeaa4cf8120b1fba5b3082daff99e951f84d285c1583ead/userdata/shm DeviceMajor:0 DeviceMinor:278 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-327 DeviceMajor:0 DeviceMinor:327 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fa82c276-5679-40f4-b7b4-e73eba7de9ef/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:229 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a225113f-cd49-48a9-b211-595bf5de763a/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fa82c276-5679-40f4-b7b4-e73eba7de9ef/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:225 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/80c66ec7-5430-41a1-8641-56db5e4e9d73/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:342 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/21eeb2fddeba4c6a0109132af0db2fdc6635c8a149fcaa1750e43d852575ed2f/userdata/shm DeviceMajor:0 DeviceMinor:352 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9553a008d6925c67e4a2b603ebe7471784b7851172e5102a19c033372f1fad59/userdata/shm DeviceMajor:0 DeviceMinor:363 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a225113f-cd49-48a9-b211-595bf5de763a/volumes/kubernetes.io~projected/kube-api-access-fs2sl DeviceMajor:0 DeviceMinor:203 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fa82c276-5679-40f4-b7b4-e73eba7de9ef/volumes/kubernetes.io~projected/kube-api-access-g7jcx DeviceMajor:0 DeviceMinor:230 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-365 DeviceMajor:0 DeviceMinor:365 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-371 DeviceMajor:0 DeviceMinor:371 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/80c66ec7-5430-41a1-8641-56db5e4e9d73/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:342 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-410 DeviceMajor:0 DeviceMinor:410 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-64 DeviceMajor:0 DeviceMinor:64 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8/volumes/kubernetes.io~projected/kube-api-access-pcwgs DeviceMajor:0 DeviceMinor:253 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-376 DeviceMajor:0 DeviceMinor:376 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5a57adad-0c11-4902-be16-b8dfb4e3ee13/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:49 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-369 DeviceMajor:0 DeviceMinor:369 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/91ef232e-99ae-42f2-be58-e17cc42f12ea/volumes/kubernetes.io~projected/kube-api-access-8z6z7 DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-367 DeviceMajor:0 DeviceMinor:367 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/40d57e7e-4896-4e0c-b266-ef5e6ca8beab/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:291 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/849c0f8d3b237ee06cab1e3c239405786fcf922f40061978e92bdd14c3ce4172/userdata/shm DeviceMajor:0 DeviceMinor:359 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-391 DeviceMajor:0 DeviceMinor:391 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/65e6a200-c6f4-4671-9d2e-b0158ff3c5ce/volumes/kubernetes.io~projected/kube-api-access-jtnhd DeviceMajor:0 DeviceMinor:51 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/40d57e7e-4896-4e0c-b266-ef5e6ca8beab/volumes/kubernetes.io~projected/kube-api-access-nq6fc DeviceMajor:0 DeviceMinor:294 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/80c66ec7-5430-41a1-8641-56db5e4e9d73/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:342 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-403 DeviceMajor:0 DeviceMinor:403 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3b6395bcdb9ea079988584faf913c7ac86c2a5cb495fcfd707a412590ea159ad/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e943b81be985021134f7b01fa18e9de2d046185a015ab296c031b2318a819812/userdata/shm DeviceMajor:0 DeviceMinor:254 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/961c6886-e297-4202-b10a-df97f3537736/volumes/kubernetes.io~projected/kube-api-access-x5bdv DeviceMajor:0 DeviceMinor:277 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f16f57a8-9630-416c-bd95-f5a2c6af2f15/volumes/kubernetes.io~projected/kube-api-access-j9xlj DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/792959e7e29e92310ac6d7f92cafef9566d8b05c6fa9973efacad40daefd3c11/userdata/shm DeviceMajor:0 DeviceMinor:295 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/80c66ec7-5430-41a1-8641-56db5e4e9d73/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:341 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:1eb923680684f91 MacAddress:da:ec:0b:fc:07:d7 Speed:10000 Mtu:8901} {Name:21eeb2fddeba4c6 MacAddress:a2:67:8e:78:e2:ed Speed:10000 Mtu:8901} {Name:3b6395bcdb9ea07 MacAddress:02:5a:84:99:7d:d6 Speed:10000 Mtu:8901} {Name:45dad8cb2d5f132 MacAddress:8e:21:34:aa:7a:af Speed:10000 Mtu:8901} {Name:56edd4df9c723fd MacAddress:72:31:83:33:50:e7 Speed:10000 Mtu:8901} {Name:792959e7e29e923 MacAddress:82:ca:c0:be:95:32 Speed:10000 Mtu:8901} {Name:849c0f8d3b237ee MacAddress:2e:54:46:6f:05:b3 Speed:10000 Mtu:8901} {Name:9553a008d6925c6 MacAddress:36:39:d9:b1:71:8d Speed:10000 Mtu:8901} {Name:b3ea44655cb46e9 MacAddress:e2:18:37:95:ee:b1 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:d5:f7:bb:6b Speed:0 Mtu:9001} {Name:br-int MacAddress:e2:b5:de:0a:d0:81 Speed:0 Mtu:8901} {Name:e7771648236c360 MacAddress:4e:27:17:f5:8f:1e Speed:10000 Mtu:8901} {Name:e943b81be985021 MacAddress:96:86:39:1e:67:4c Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:d5:f7:bb:6b Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:ba:db:94:cc:7c:a0 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:da:f6:0f:61:46:b8 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 01 19:59:14.298767 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.294883 5935 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 01 19:59:14.298767 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.294932 5935 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 01 19:59:14.298767 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.295478 5935 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 01 19:59:14.298767 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.295506 5935 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-26.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 01 19:59:14.298767 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.295823 5935 topology_manager.go:143] "Creating topology manager with none policy" Nov 01 19:59:14.298767 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.295835 5935 container_manager_linux.go:311] "Creating device plugin manager" Nov 01 19:59:14.298767 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.295849 5935 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 19:59:14.298767 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.295867 5935 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 01 19:59:14.298767 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.296012 5935 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 01 19:59:14.298767 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.296342 5935 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 01 19:59:14.298767 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.296684 5935 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 01 19:59:14.298767 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.296729 5935 kubelet.go:493] "Attempting to sync node with API server" Nov 01 19:59:14.298767 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.296749 5935 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 01 19:59:14.298767 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.296762 5935 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 01 19:59:14.298767 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.296772 5935 kubelet.go:405] "Adding apiserver pod source" Nov 01 19:59:14.298767 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.296780 5935 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 01 19:59:14.298767 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.297467 5935 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 01 19:59:14.298767 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.297478 5935 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 19:59:14.298767 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.297988 5935 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 01 19:59:14.300399 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.300386 5935 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 01 19:59:14.300399 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.300399 5935 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 01 19:59:14.300500 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.300490 5935 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 01 19:59:14.300673 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.300663 5935 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 01 19:59:14.300737 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.300692 5935 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 01 19:59:14.300787 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.300779 5935 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 01 19:59:14.300816 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.300793 5935 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 01 19:59:14.300816 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.300799 5935 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 01 19:59:14.300816 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.300804 5935 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 01 19:59:14.300816 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.300810 5935 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 01 19:59:14.300816 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.300816 5935 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 01 19:59:14.300952 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.300822 5935 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 01 19:59:14.300952 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.300827 5935 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 01 19:59:14.300952 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.300835 5935 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 01 19:59:14.300952 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.300840 5935 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 01 19:59:14.300952 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.300855 5935 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 01 19:59:14.300952 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.300864 5935 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 01 19:59:14.300952 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.300889 5935 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 01 19:59:14.300952 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.300894 5935 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 01 19:59:14.304427 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.304412 5935 server.go:1263] "Started kubelet" Nov 01 19:59:14.304543 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.304516 5935 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 01 19:59:14.304638 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.304575 5935 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 01 19:59:14.304944 ip-10-0-136-26 systemd[1]: Started Kubernetes Kubelet. Nov 01 19:59:14.305050 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.305015 5935 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 01 19:59:14.305844 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.305731 5935 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 01 19:59:14.306926 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.306214 5935 server.go:317] "Adding debug handlers to kubelet server" Nov 01 19:59:14.310990 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.310953 5935 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 01 19:59:14.311139 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.310955 5935 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 01 19:59:14.311663 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.311401 5935 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 01 19:59:14.311663 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.311419 5935 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 01 19:59:14.311663 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.311402 5935 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 01 19:59:14.311663 ip-10-0-136-26 kubenswrapper[5935]: E1101 19:59:14.311379 5935 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-26.us-east-2.compute.internal\" not found" Nov 01 19:59:14.311966 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.311948 5935 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 19:59:14.312131 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.312106 5935 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 19:53:03 +0000 UTC" deadline="2028-06-28 17:41:23.796239344 +0000 UTC" Nov 01 19:59:14.312524 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.312456 5935 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14517h42m9.483791657s" Nov 01 19:59:14.313167 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.313149 5935 factory.go:153] Registering CRI-O factory Nov 01 19:59:14.313167 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.313168 5935 factory.go:223] Registration of the crio container factory successfully Nov 01 19:59:14.315125 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.313221 5935 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 01 19:59:14.315125 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.313229 5935 factory.go:55] Registering systemd factory Nov 01 19:59:14.315125 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.313239 5935 factory.go:223] Registration of the systemd container factory successfully Nov 01 19:59:14.315125 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.313256 5935 factory.go:103] Registering Raw factory Nov 01 19:59:14.315125 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.313270 5935 manager.go:1199] Started watching for new ooms in manager Nov 01 19:59:14.315125 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.313808 5935 manager.go:319] Starting recovery of all containers Nov 01 19:59:14.315435 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.315218 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40d57e7e-4896-4e0c-b266-ef5e6ca8beab" volumeName="kubernetes.io/projected/40d57e7e-4896-4e0c-b266-ef5e6ca8beab-kube-api-access-nq6fc" seLinuxMountContext="" Nov 01 19:59:14.315435 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.315266 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80c66ec7-5430-41a1-8641-56db5e4e9d73" volumeName="kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Nov 01 19:59:14.315435 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.315299 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83a0642b-4dfe-4ddb-b506-f0654c7b6d30" volumeName="kubernetes.io/projected/83a0642b-4dfe-4ddb-b506-f0654c7b6d30-kube-api-access-bm8bk" seLinuxMountContext="" Nov 01 19:59:14.315435 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.315313 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40d57e7e-4896-4e0c-b266-ef5e6ca8beab" volumeName="kubernetes.io/configmap/40d57e7e-4896-4e0c-b266-ef5e6ca8beab-metrics-client-ca" seLinuxMountContext="" Nov 01 19:59:14.315435 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.315325 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58167352-076f-40d7-abee-1c5f48f67566" volumeName="kubernetes.io/configmap/58167352-076f-40d7-abee-1c5f48f67566-config-volume" seLinuxMountContext="" Nov 01 19:59:14.315435 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.315338 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83a0642b-4dfe-4ddb-b506-f0654c7b6d30" volumeName="kubernetes.io/configmap/83a0642b-4dfe-4ddb-b506-f0654c7b6d30-kube-rbac-proxy-cm" seLinuxMountContext="" Nov 01 19:59:14.315435 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.315352 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="961c6886-e297-4202-b10a-df97f3537736" volumeName="kubernetes.io/projected/961c6886-e297-4202-b10a-df97f3537736-kube-api-access-x5bdv" seLinuxMountContext="" Nov 01 19:59:14.315435 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.315364 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="983a921d-61a5-4668-8e17-1a24b6eaf8b6" volumeName="kubernetes.io/configmap/983a921d-61a5-4668-8e17-1a24b6eaf8b6-serviceca" seLinuxMountContext="" Nov 01 19:59:14.315435 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.315390 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e564268-fa85-48e6-8d1c-5069cc0f9648" volumeName="kubernetes.io/projected/9e564268-fa85-48e6-8d1c-5069cc0f9648-kube-api-access-mnjs4" seLinuxMountContext="" Nov 01 19:59:14.315435 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.315402 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0e3a247-a792-431d-85cb-9b46982c3373" volumeName="kubernetes.io/projected/c0e3a247-a792-431d-85cb-9b46982c3373-kube-api-access-nz4mx" seLinuxMountContext="" Nov 01 19:59:14.315435 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.315412 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f16f57a8-9630-416c-bd95-f5a2c6af2f15" volumeName="kubernetes.io/configmap/f16f57a8-9630-416c-bd95-f5a2c6af2f15-iptables-alerter-script" seLinuxMountContext="" Nov 01 19:59:14.315435 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.315425 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58167352-076f-40d7-abee-1c5f48f67566" volumeName="kubernetes.io/projected/58167352-076f-40d7-abee-1c5f48f67566-kube-api-access-chtj4" seLinuxMountContext="" Nov 01 19:59:14.315435 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.315441 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80c66ec7-5430-41a1-8641-56db5e4e9d73" volumeName="kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-secret-metrics-client-certs" seLinuxMountContext="" Nov 01 19:59:14.316156 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.315454 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80c66ec7-5430-41a1-8641-56db5e4e9d73" volumeName="kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-secret-kube-rbac-proxy" seLinuxMountContext="" Nov 01 19:59:14.316156 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.315469 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83a0642b-4dfe-4ddb-b506-f0654c7b6d30" volumeName="kubernetes.io/secret/83a0642b-4dfe-4ddb-b506-f0654c7b6d30-insights-runtime-extractor-tls" seLinuxMountContext="" Nov 01 19:59:14.316156 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.315482 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87d7d5b7-5e96-4777-afcb-2a26d45b0ba8" volumeName="kubernetes.io/configmap/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-service-ca" seLinuxMountContext="" Nov 01 19:59:14.316156 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.315508 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fa82c276-5679-40f4-b7b4-e73eba7de9ef" volumeName="kubernetes.io/projected/fa82c276-5679-40f4-b7b4-e73eba7de9ef-kube-api-access-g7jcx" seLinuxMountContext="" Nov 01 19:59:14.316156 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.315519 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1cd0e9b2-272f-46f0-88d3-89f58c5dab49" volumeName="kubernetes.io/configmap/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-env-overrides" seLinuxMountContext="" Nov 01 19:59:14.316156 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.315533 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1cd0e9b2-272f-46f0-88d3-89f58c5dab49" volumeName="kubernetes.io/configmap/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-ovnkube-script-lib" seLinuxMountContext="" Nov 01 19:59:14.316156 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.315546 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65e6a200-c6f4-4671-9d2e-b0158ff3c5ce" volumeName="kubernetes.io/projected/65e6a200-c6f4-4671-9d2e-b0158ff3c5ce-kube-api-access-jtnhd" seLinuxMountContext="" Nov 01 19:59:14.316156 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.315559 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80c66ec7-5430-41a1-8641-56db5e4e9d73" volumeName="kubernetes.io/projected/80c66ec7-5430-41a1-8641-56db5e4e9d73-kube-api-access-x4zhc" seLinuxMountContext="" Nov 01 19:59:14.316156 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.315570 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f16f57a8-9630-416c-bd95-f5a2c6af2f15" volumeName="kubernetes.io/projected/f16f57a8-9630-416c-bd95-f5a2c6af2f15-kube-api-access-j9xlj" seLinuxMountContext="" Nov 01 19:59:14.316156 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.315581 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40d57e7e-4896-4e0c-b266-ef5e6ca8beab" volumeName="kubernetes.io/empty-dir/40d57e7e-4896-4e0c-b266-ef5e6ca8beab-tmp" seLinuxMountContext="" Nov 01 19:59:14.316156 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.315596 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60e3a171-27e7-4ae6-ad6c-89984ecf5634" volumeName="kubernetes.io/projected/60e3a171-27e7-4ae6-ad6c-89984ecf5634-kube-api-access-4cq6h" seLinuxMountContext="" Nov 01 19:59:14.316156 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.315611 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80c66ec7-5430-41a1-8641-56db5e4e9d73" volumeName="kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-web-config" seLinuxMountContext="" Nov 01 19:59:14.316156 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.315622 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83a0642b-4dfe-4ddb-b506-f0654c7b6d30" volumeName="kubernetes.io/empty-dir/83a0642b-4dfe-4ddb-b506-f0654c7b6d30-data-volume" seLinuxMountContext="" Nov 01 19:59:14.316156 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.315637 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="961c6886-e297-4202-b10a-df97f3537736" volumeName="kubernetes.io/configmap/961c6886-e297-4202-b10a-df97f3537736-node-exporter-accelerators-collector-config" seLinuxMountContext="" Nov 01 19:59:14.316156 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.315650 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58167352-076f-40d7-abee-1c5f48f67566" volumeName="kubernetes.io/secret/58167352-076f-40d7-abee-1c5f48f67566-metrics-tls" seLinuxMountContext="" Nov 01 19:59:14.316156 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.315665 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60e3a171-27e7-4ae6-ad6c-89984ecf5634" volumeName="kubernetes.io/configmap/60e3a171-27e7-4ae6-ad6c-89984ecf5634-trusted-ca-bundle" seLinuxMountContext="" Nov 01 19:59:14.316156 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.315678 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62c9da55-b21d-4f8f-a3ec-98e823655351" volumeName="kubernetes.io/configmap/62c9da55-b21d-4f8f-a3ec-98e823655351-cni-binary-copy" seLinuxMountContext="" Nov 01 19:59:14.316156 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.315692 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="65e6a200-c6f4-4671-9d2e-b0158ff3c5ce" volumeName="kubernetes.io/secret/65e6a200-c6f4-4671-9d2e-b0158ff3c5ce-metrics-certs" seLinuxMountContext="" Nov 01 19:59:14.316156 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.315707 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80c66ec7-5430-41a1-8641-56db5e4e9d73" volumeName="kubernetes.io/configmap/80c66ec7-5430-41a1-8641-56db5e4e9d73-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Nov 01 19:59:14.316156 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.315738 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80c66ec7-5430-41a1-8641-56db5e4e9d73" volumeName="kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-config" seLinuxMountContext="" Nov 01 19:59:14.316156 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.315754 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2efc5bbf-efd4-4a42-bed8-53f2490bb7bc" volumeName="kubernetes.io/secret/2efc5bbf-efd4-4a42-bed8-53f2490bb7bc-original-pull-secret" seLinuxMountContext="" Nov 01 19:59:14.316156 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.315766 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40d57e7e-4896-4e0c-b266-ef5e6ca8beab" volumeName="kubernetes.io/secret/40d57e7e-4896-4e0c-b266-ef5e6ca8beab-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Nov 01 19:59:14.316156 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.315778 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62c9da55-b21d-4f8f-a3ec-98e823655351" volumeName="kubernetes.io/projected/62c9da55-b21d-4f8f-a3ec-98e823655351-kube-api-access-fhv76" seLinuxMountContext="" Nov 01 19:59:14.316156 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.315788 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80c66ec7-5430-41a1-8641-56db5e4e9d73" volumeName="kubernetes.io/configmap/80c66ec7-5430-41a1-8641-56db5e4e9d73-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Nov 01 19:59:14.316156 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.315799 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80c66ec7-5430-41a1-8641-56db5e4e9d73" volumeName="kubernetes.io/configmap/80c66ec7-5430-41a1-8641-56db5e4e9d73-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Nov 01 19:59:14.316156 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.315811 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80c66ec7-5430-41a1-8641-56db5e4e9d73" volumeName="kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-secret-prometheus-k8s-tls" seLinuxMountContext="" Nov 01 19:59:14.316156 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.315826 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87d7d5b7-5e96-4777-afcb-2a26d45b0ba8" volumeName="kubernetes.io/configmap/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-oauth-serving-cert" seLinuxMountContext="" Nov 01 19:59:14.316156 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.315840 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1cd0e9b2-272f-46f0-88d3-89f58c5dab49" volumeName="kubernetes.io/projected/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-kube-api-access-6dg6b" seLinuxMountContext="" Nov 01 19:59:14.316156 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.315854 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80c66ec7-5430-41a1-8641-56db5e4e9d73" volumeName="kubernetes.io/configmap/80c66ec7-5430-41a1-8641-56db5e4e9d73-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Nov 01 19:59:14.316156 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.315867 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8537e78c-5b2c-4966-9558-044ddc39119b" volumeName="kubernetes.io/configmap/8537e78c-5b2c-4966-9558-044ddc39119b-konnectivity-ca" seLinuxMountContext="" Nov 01 19:59:14.316156 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.315878 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8537e78c-5b2c-4966-9558-044ddc39119b" volumeName="kubernetes.io/secret/8537e78c-5b2c-4966-9558-044ddc39119b-agent-certs" seLinuxMountContext="" Nov 01 19:59:14.316156 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.315891 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="983a921d-61a5-4668-8e17-1a24b6eaf8b6" volumeName="kubernetes.io/projected/983a921d-61a5-4668-8e17-1a24b6eaf8b6-kube-api-access-8fntm" seLinuxMountContext="" Nov 01 19:59:14.316156 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.315904 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fa82c276-5679-40f4-b7b4-e73eba7de9ef" volumeName="kubernetes.io/configmap/fa82c276-5679-40f4-b7b4-e73eba7de9ef-metrics-client-ca" seLinuxMountContext="" Nov 01 19:59:14.316156 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.315918 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fa82c276-5679-40f4-b7b4-e73eba7de9ef" volumeName="kubernetes.io/secret/fa82c276-5679-40f4-b7b4-e73eba7de9ef-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext="" Nov 01 19:59:14.316156 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.315933 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fa82c276-5679-40f4-b7b4-e73eba7de9ef" volumeName="kubernetes.io/secret/fa82c276-5679-40f4-b7b4-e73eba7de9ef-prometheus-operator-tls" seLinuxMountContext="" Nov 01 19:59:14.316156 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.315944 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40d57e7e-4896-4e0c-b266-ef5e6ca8beab" volumeName="kubernetes.io/secret/40d57e7e-4896-4e0c-b266-ef5e6ca8beab-secret-grpc-tls" seLinuxMountContext="" Nov 01 19:59:14.316156 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.315955 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58167352-076f-40d7-abee-1c5f48f67566" volumeName="kubernetes.io/empty-dir/58167352-076f-40d7-abee-1c5f48f67566-tmp-dir" seLinuxMountContext="" Nov 01 19:59:14.316156 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.315970 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80c66ec7-5430-41a1-8641-56db5e4e9d73" volumeName="kubernetes.io/projected/80c66ec7-5430-41a1-8641-56db5e4e9d73-tls-assets" seLinuxMountContext="" Nov 01 19:59:14.316156 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.315981 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87d7d5b7-5e96-4777-afcb-2a26d45b0ba8" volumeName="kubernetes.io/configmap/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-trusted-ca-bundle" seLinuxMountContext="" Nov 01 19:59:14.316156 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.315993 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="91ef232e-99ae-42f2-be58-e17cc42f12ea" volumeName="kubernetes.io/projected/91ef232e-99ae-42f2-be58-e17cc42f12ea-kube-api-access-8z6z7" seLinuxMountContext="" Nov 01 19:59:14.316156 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.316026 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87d7d5b7-5e96-4777-afcb-2a26d45b0ba8" volumeName="kubernetes.io/empty-dir/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-tmp" seLinuxMountContext="" Nov 01 19:59:14.316156 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.316040 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87d7d5b7-5e96-4777-afcb-2a26d45b0ba8" volumeName="kubernetes.io/projected/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-kube-api-access-pcwgs" seLinuxMountContext="" Nov 01 19:59:14.316156 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.316053 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87d7d5b7-5e96-4777-afcb-2a26d45b0ba8" volumeName="kubernetes.io/secret/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-console-serving-cert" seLinuxMountContext="" Nov 01 19:59:14.316156 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.316077 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="961c6886-e297-4202-b10a-df97f3537736" volumeName="kubernetes.io/configmap/961c6886-e297-4202-b10a-df97f3537736-metrics-client-ca" seLinuxMountContext="" Nov 01 19:59:14.316156 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.316093 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="961c6886-e297-4202-b10a-df97f3537736" volumeName="kubernetes.io/secret/961c6886-e297-4202-b10a-df97f3537736-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Nov 01 19:59:14.316156 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.316105 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60e3a171-27e7-4ae6-ad6c-89984ecf5634" volumeName="kubernetes.io/configmap/60e3a171-27e7-4ae6-ad6c-89984ecf5634-console-config" seLinuxMountContext="" Nov 01 19:59:14.316156 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.316123 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80c66ec7-5430-41a1-8641-56db5e4e9d73" volumeName="kubernetes.io/empty-dir/80c66ec7-5430-41a1-8641-56db5e4e9d73-prometheus-k8s-db" seLinuxMountContext="" Nov 01 19:59:14.316156 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.316137 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80c66ec7-5430-41a1-8641-56db5e4e9d73" volumeName="kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-secret-grpc-tls" seLinuxMountContext="" Nov 01 19:59:14.316156 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.316149 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e564268-fa85-48e6-8d1c-5069cc0f9648" volumeName="kubernetes.io/configmap/9e564268-fa85-48e6-8d1c-5069cc0f9648-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 01 19:59:14.316156 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.316168 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e564268-fa85-48e6-8d1c-5069cc0f9648" volumeName="kubernetes.io/configmap/9e564268-fa85-48e6-8d1c-5069cc0f9648-cni-sysctl-allowlist" seLinuxMountContext="" Nov 01 19:59:14.317741 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.316193 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a225113f-cd49-48a9-b211-595bf5de763a" volumeName="kubernetes.io/secret/a225113f-cd49-48a9-b211-595bf5de763a-cert" seLinuxMountContext="" Nov 01 19:59:14.317741 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.316205 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1cd0e9b2-272f-46f0-88d3-89f58c5dab49" volumeName="kubernetes.io/secret/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-ovn-node-metrics-cert" seLinuxMountContext="" Nov 01 19:59:14.317741 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.316228 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60e3a171-27e7-4ae6-ad6c-89984ecf5634" volumeName="kubernetes.io/configmap/60e3a171-27e7-4ae6-ad6c-89984ecf5634-oauth-serving-cert" seLinuxMountContext="" Nov 01 19:59:14.317741 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.316240 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60e3a171-27e7-4ae6-ad6c-89984ecf5634" volumeName="kubernetes.io/secret/60e3a171-27e7-4ae6-ad6c-89984ecf5634-console-serving-cert" seLinuxMountContext="" Nov 01 19:59:14.317741 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.316250 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80c66ec7-5430-41a1-8641-56db5e4e9d73" volumeName="kubernetes.io/configmap/80c66ec7-5430-41a1-8641-56db5e4e9d73-configmap-metrics-client-ca" seLinuxMountContext="" Nov 01 19:59:14.317741 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.316262 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80c66ec7-5430-41a1-8641-56db5e4e9d73" volumeName="kubernetes.io/configmap/80c66ec7-5430-41a1-8641-56db5e4e9d73-prometheus-trusted-ca-bundle" seLinuxMountContext="" Nov 01 19:59:14.317741 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.316297 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e564268-fa85-48e6-8d1c-5069cc0f9648" volumeName="kubernetes.io/configmap/9e564268-fa85-48e6-8d1c-5069cc0f9648-cni-binary-copy" seLinuxMountContext="" Nov 01 19:59:14.317741 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.315326 5935 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 19:59:14.317741 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.316310 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40d57e7e-4896-4e0c-b266-ef5e6ca8beab" volumeName="kubernetes.io/secret/40d57e7e-4896-4e0c-b266-ef5e6ca8beab-secret-thanos-querier-tls" seLinuxMountContext="" Nov 01 19:59:14.317741 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.316333 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a57adad-0c11-4902-be16-b8dfb4e3ee13" volumeName="kubernetes.io/projected/5a57adad-0c11-4902-be16-b8dfb4e3ee13-kube-api-access-98mhk" seLinuxMountContext="" Nov 01 19:59:14.317741 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.316347 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80c66ec7-5430-41a1-8641-56db5e4e9d73" volumeName="kubernetes.io/configmap/80c66ec7-5430-41a1-8641-56db5e4e9d73-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Nov 01 19:59:14.317741 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.316369 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87d7d5b7-5e96-4777-afcb-2a26d45b0ba8" volumeName="kubernetes.io/configmap/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-console-config" seLinuxMountContext="" Nov 01 19:59:14.317741 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.316380 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="961c6886-e297-4202-b10a-df97f3537736" volumeName="kubernetes.io/empty-dir/961c6886-e297-4202-b10a-df97f3537736-node-exporter-textfile" seLinuxMountContext="" Nov 01 19:59:14.317741 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.316392 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0e3a247-a792-431d-85cb-9b46982c3373" volumeName="kubernetes.io/empty-dir/c0e3a247-a792-431d-85cb-9b46982c3373-tmp-dir" seLinuxMountContext="" Nov 01 19:59:14.317741 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.316403 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80c66ec7-5430-41a1-8641-56db5e4e9d73" volumeName="kubernetes.io/empty-dir/80c66ec7-5430-41a1-8641-56db5e4e9d73-config-out" seLinuxMountContext="" Nov 01 19:59:14.317741 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.316415 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80c66ec7-5430-41a1-8641-56db5e4e9d73" volumeName="kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Nov 01 19:59:14.317741 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.316426 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="16e51d04-90d5-484d-9f85-e9d484205220" volumeName="kubernetes.io/projected/16e51d04-90d5-484d-9f85-e9d484205220-kube-api-access-vq4k2" seLinuxMountContext="" Nov 01 19:59:14.317741 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.316437 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40d57e7e-4896-4e0c-b266-ef5e6ca8beab" volumeName="kubernetes.io/secret/40d57e7e-4896-4e0c-b266-ef5e6ca8beab-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Nov 01 19:59:14.317741 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.316448 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a57adad-0c11-4902-be16-b8dfb4e3ee13" volumeName="kubernetes.io/empty-dir/5a57adad-0c11-4902-be16-b8dfb4e3ee13-tmp" seLinuxMountContext="" Nov 01 19:59:14.317741 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.316457 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87d7d5b7-5e96-4777-afcb-2a26d45b0ba8" volumeName="kubernetes.io/secret/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-console-oauth-config" seLinuxMountContext="" Nov 01 19:59:14.317741 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.316466 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="961c6886-e297-4202-b10a-df97f3537736" volumeName="kubernetes.io/secret/961c6886-e297-4202-b10a-df97f3537736-node-exporter-tls" seLinuxMountContext="" Nov 01 19:59:14.317741 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.316482 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a225113f-cd49-48a9-b211-595bf5de763a" volumeName="kubernetes.io/projected/a225113f-cd49-48a9-b211-595bf5de763a-kube-api-access-fs2sl" seLinuxMountContext="" Nov 01 19:59:14.317741 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.316490 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40d57e7e-4896-4e0c-b266-ef5e6ca8beab" volumeName="kubernetes.io/secret/40d57e7e-4896-4e0c-b266-ef5e6ca8beab-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Nov 01 19:59:14.317741 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.316497 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40d57e7e-4896-4e0c-b266-ef5e6ca8beab" volumeName="kubernetes.io/secret/40d57e7e-4896-4e0c-b266-ef5e6ca8beab-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Nov 01 19:59:14.317741 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.316505 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a57adad-0c11-4902-be16-b8dfb4e3ee13" volumeName="kubernetes.io/empty-dir/5a57adad-0c11-4902-be16-b8dfb4e3ee13-etc-tuned" seLinuxMountContext="" Nov 01 19:59:14.317741 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.316513 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60e3a171-27e7-4ae6-ad6c-89984ecf5634" volumeName="kubernetes.io/configmap/60e3a171-27e7-4ae6-ad6c-89984ecf5634-service-ca" seLinuxMountContext="" Nov 01 19:59:14.317741 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.316520 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60e3a171-27e7-4ae6-ad6c-89984ecf5634" volumeName="kubernetes.io/empty-dir/60e3a171-27e7-4ae6-ad6c-89984ecf5634-tmp" seLinuxMountContext="" Nov 01 19:59:14.317741 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.316528 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="60e3a171-27e7-4ae6-ad6c-89984ecf5634" volumeName="kubernetes.io/secret/60e3a171-27e7-4ae6-ad6c-89984ecf5634-console-oauth-config" seLinuxMountContext="" Nov 01 19:59:14.317741 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.316535 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62c9da55-b21d-4f8f-a3ec-98e823655351" volumeName="kubernetes.io/configmap/62c9da55-b21d-4f8f-a3ec-98e823655351-multus-daemon-config" seLinuxMountContext="" Nov 01 19:59:14.317741 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.316542 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="80c66ec7-5430-41a1-8641-56db5e4e9d73" volumeName="kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-thanos-prometheus-http-client-file" seLinuxMountContext="" Nov 01 19:59:14.317741 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.316552 5935 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1cd0e9b2-272f-46f0-88d3-89f58c5dab49" volumeName="kubernetes.io/configmap/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-ovnkube-config" seLinuxMountContext="" Nov 01 19:59:14.317741 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.316560 5935 reconstruct.go:97] "Volume reconstruction finished" Nov 01 19:59:14.317741 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.316565 5935 reconciler.go:29] "Reconciler: start to sync state" Nov 01 19:59:14.317741 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.316964 5935 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 19:59:14.319266 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.319202 5935 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 01 19:59:14.344120 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.344059 5935 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 01 19:59:14.345586 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.345559 5935 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 01 19:59:14.345586 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.345583 5935 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 01 19:59:14.345774 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.345608 5935 kubelet.go:2521] "Starting kubelet main sync loop" Nov 01 19:59:14.345774 ip-10-0-136-26 kubenswrapper[5935]: E1101 19:59:14.345653 5935 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 01 19:59:14.349503 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.349479 5935 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 01 19:59:14.366726 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.366677 5935 generic.go:356] "Generic (PLEG): container finished" podID="80c66ec7-5430-41a1-8641-56db5e4e9d73" containerID="86c3f1c47b655a4accd6f8c872ff24bcfd277f8e89dd328467251033cab7562f" exitCode=0 Nov 01 19:59:14.368789 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.368760 5935 generic.go:356] "Generic (PLEG): container finished" podID="961c6886-e297-4202-b10a-df97f3537736" containerID="9c0c31ce88da86d198c89708d2756542617d5bbb78f87a0b31240e9be1ad536f" exitCode=0 Nov 01 19:59:14.372349 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.372325 5935 generic.go:356] "Generic (PLEG): container finished" podID="7f0999f9286775a451886a78f6ad439c" containerID="7b56774f13c2ea90da8bdcd69d2446d18dca5504b87d7df239bde9576c234850" exitCode=0 Nov 01 19:59:14.381631 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.381596 5935 generic.go:356] "Generic (PLEG): container finished" podID="9e564268-fa85-48e6-8d1c-5069cc0f9648" containerID="1d2d55a79a113336883bde47f67a5a39c2b981bc4213b0b217df4ac153356c63" exitCode=0 Nov 01 19:59:14.381631 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.381631 5935 generic.go:356] "Generic (PLEG): container finished" podID="9e564268-fa85-48e6-8d1c-5069cc0f9648" containerID="80aecb968da642626a5cbe71a2c9b65ba6cdd5d5f71a17143b7ad340682d5243" exitCode=0 Nov 01 19:59:14.381631 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.381644 5935 generic.go:356] "Generic (PLEG): container finished" podID="9e564268-fa85-48e6-8d1c-5069cc0f9648" containerID="f28392decd350e3ab40ca8de6b94d98606e733e9f2029863759e4bb32a6403f8" exitCode=0 Nov 01 19:59:14.381869 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.381660 5935 generic.go:356] "Generic (PLEG): container finished" podID="9e564268-fa85-48e6-8d1c-5069cc0f9648" containerID="f04c5c5dd003201e88f0c916ccc81b38981e72a118a47cc080c7b7aee0c8bff6" exitCode=0 Nov 01 19:59:14.381869 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.381670 5935 generic.go:356] "Generic (PLEG): container finished" podID="9e564268-fa85-48e6-8d1c-5069cc0f9648" containerID="d94a19b39b1dcc753a69bcbbcf68a943d04d75dd342964056f010d82b9d30ab7" exitCode=0 Nov 01 19:59:14.381869 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.381682 5935 generic.go:356] "Generic (PLEG): container finished" podID="9e564268-fa85-48e6-8d1c-5069cc0f9648" containerID="cbb74d3f8ffe788c98c5d23d01635c377f6e2ce2056c44594e1fc70b46bd9df1" exitCode=0 Nov 01 19:59:14.417147 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.417095 5935 manager.go:324] Recovery completed Nov 01 19:59:14.438638 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.438612 5935 cpu_manager.go:226] "Starting" policy="none" Nov 01 19:59:14.438638 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.438630 5935 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 01 19:59:14.438785 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.438648 5935 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 01 19:59:14.438785 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.438761 5935 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 01 19:59:14.438785 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.438772 5935 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 01 19:59:14.438897 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.438788 5935 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 01 19:59:14.438897 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.438793 5935 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 01 19:59:14.438897 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.438798 5935 policy_none.go:50] "Start" Nov 01 19:59:14.438897 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.438807 5935 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 01 19:59:14.438897 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.438816 5935 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 01 19:59:14.439030 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.438936 5935 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 01 19:59:14.439030 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.438943 5935 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 01 19:59:14.439030 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.438948 5935 policy_none.go:44] "Start" Nov 01 19:59:14.444513 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.444484 5935 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 01 19:59:14.446193 ip-10-0-136-26 kubenswrapper[5935]: E1101 19:59:14.446174 5935 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 01 19:59:14.448038 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.448025 5935 manager.go:341] "Starting Device Plugin manager" Nov 01 19:59:14.448514 ip-10-0-136-26 kubenswrapper[5935]: E1101 19:59:14.448053 5935 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 01 19:59:14.448514 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.448518 5935 server.go:87] "Starting device plugin registration server" Nov 01 19:59:14.448810 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.448796 5935 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 01 19:59:14.448875 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.448810 5935 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 01 19:59:14.449021 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.448878 5935 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 01 19:59:14.449021 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.448969 5935 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 01 19:59:14.449021 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.449005 5935 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 01 19:59:14.449021 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.449012 5935 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 01 19:59:14.449201 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.449062 5935 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T19:59:14.44899565Z","UUID":"67791c8e-57ea-46b1-873b-1985c10fb273","Handler":null,"Name":"","Endpoint":""} Nov 01 19:59:14.450886 ip-10-0-136-26 kubenswrapper[5935]: E1101 19:59:14.450859 5935 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 01 19:59:14.451105 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.451088 5935 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 01 19:59:14.451181 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.451125 5935 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 01 19:59:14.549615 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.549593 5935 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 01 19:59:14.551440 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.551423 5935 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-26.us-east-2.compute.internal" event="NodeHasSufficientMemory" Nov 01 19:59:14.551514 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.551451 5935 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-26.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Nov 01 19:59:14.551514 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.551461 5935 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-26.us-east-2.compute.internal" event="NodeHasSufficientPID" Nov 01 19:59:14.551514 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.551505 5935 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-136-26.us-east-2.compute.internal" Nov 01 19:59:14.566369 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.566352 5935 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-136-26.us-east-2.compute.internal" Nov 01 19:59:14.566463 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.566412 5935 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-136-26.us-east-2.compute.internal" Nov 01 19:59:14.647156 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.647114 5935 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-26.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-26.us-east-2.compute.internal"] Nov 01 19:59:14.647795 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.647770 5935 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-26.us-east-2.compute.internal" Nov 01 19:59:14.647862 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.647751 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-26.us-east-2.compute.internal" event={"ID":"7f0999f9286775a451886a78f6ad439c","Type":"ContainerStarted","Data":"d4a6e4405db18f3a12663796380fdbecf6908bb5c7979df9969f18f942132ff6"} Nov 01 19:59:14.647938 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.647887 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-26.us-east-2.compute.internal" event={"ID":"7f0999f9286775a451886a78f6ad439c","Type":"ContainerDied","Data":"7b56774f13c2ea90da8bdcd69d2446d18dca5504b87d7df239bde9576c234850"} Nov 01 19:59:14.647938 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.647913 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-26.us-east-2.compute.internal" event={"ID":"7f0999f9286775a451886a78f6ad439c","Type":"ContainerStarted","Data":"23796b1cbe6433b0a4bdabc415c684b57ffbafc67eda1d4bbfd8bb577981ba61"} Nov 01 19:59:14.648036 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.647946 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-26.us-east-2.compute.internal" event={"ID":"d535588bb4025d791b22a311a0d6e9f5","Type":"ContainerStarted","Data":"bda95036bab0ac9ac16630ae43929f8d6eb3cd2ab1f16a6c01b8fdb8c1beaccc"} Nov 01 19:59:14.648036 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.647965 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-26.us-east-2.compute.internal" event={"ID":"d535588bb4025d791b22a311a0d6e9f5","Type":"ContainerStarted","Data":"0b5b365de0b6be1ee91aec65b04995e35d9cccda934e4cb6491bcde0f1bcf0b0"} Nov 01 19:59:14.648036 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.647996 5935 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-26.us-east-2.compute.internal" Nov 01 19:59:14.660841 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.660819 5935 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 19:59:14.660924 ip-10-0-136-26 kubenswrapper[5935]: E1101 19:59:14.660861 5935 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-136-26.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-26.us-east-2.compute.internal" Nov 01 19:59:14.661481 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.661469 5935 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 19:59:14.661517 ip-10-0-136-26 kubenswrapper[5935]: E1101 19:59:14.661501 5935 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-136-26.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-26.us-east-2.compute.internal" Nov 01 19:59:14.720784 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.720760 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7f0999f9286775a451886a78f6ad439c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-26.us-east-2.compute.internal\" (UID: \"7f0999f9286775a451886a78f6ad439c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-26.us-east-2.compute.internal" Nov 01 19:59:14.720869 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.720787 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f0999f9286775a451886a78f6ad439c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-26.us-east-2.compute.internal\" (UID: \"7f0999f9286775a451886a78f6ad439c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-26.us-east-2.compute.internal" Nov 01 19:59:14.720869 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.720806 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d535588bb4025d791b22a311a0d6e9f5-config\") pod \"kube-apiserver-proxy-ip-10-0-136-26.us-east-2.compute.internal\" (UID: \"d535588bb4025d791b22a311a0d6e9f5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-26.us-east-2.compute.internal" Nov 01 19:59:14.821353 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.821326 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d535588bb4025d791b22a311a0d6e9f5-config\") pod \"kube-apiserver-proxy-ip-10-0-136-26.us-east-2.compute.internal\" (UID: \"d535588bb4025d791b22a311a0d6e9f5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-26.us-east-2.compute.internal" Nov 01 19:59:14.821353 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.821354 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7f0999f9286775a451886a78f6ad439c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-26.us-east-2.compute.internal\" (UID: \"7f0999f9286775a451886a78f6ad439c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-26.us-east-2.compute.internal" Nov 01 19:59:14.821475 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.821371 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f0999f9286775a451886a78f6ad439c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-26.us-east-2.compute.internal\" (UID: \"7f0999f9286775a451886a78f6ad439c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-26.us-east-2.compute.internal" Nov 01 19:59:14.821475 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.821409 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f0999f9286775a451886a78f6ad439c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-26.us-east-2.compute.internal\" (UID: \"7f0999f9286775a451886a78f6ad439c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-26.us-east-2.compute.internal" Nov 01 19:59:14.821475 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.821423 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d535588bb4025d791b22a311a0d6e9f5-config\") pod \"kube-apiserver-proxy-ip-10-0-136-26.us-east-2.compute.internal\" (UID: \"d535588bb4025d791b22a311a0d6e9f5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-26.us-east-2.compute.internal" Nov 01 19:59:14.821475 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:14.821437 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7f0999f9286775a451886a78f6ad439c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-26.us-east-2.compute.internal\" (UID: \"7f0999f9286775a451886a78f6ad439c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-26.us-east-2.compute.internal" Nov 01 19:59:15.297382 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.297350 5935 apiserver.go:52] "Watching apiserver" Nov 01 19:59:15.309704 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.309677 5935 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 01 19:59:15.310303 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.310261 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-rz7h2","openshift-network-diagnostics/network-check-target-b7mcz","kube-system/kube-apiserver-proxy-ip-10-0-136-26.us-east-2.compute.internal","openshift-ingress-canary/ingress-canary-tq7l5","openshift-console/console-865bfb674b-jlr46","kube-system/global-pull-secret-syncer-k5brd","openshift-dns/dns-default-5lztx","openshift-ovn-kubernetes/ovnkube-node-cffbl","openshift-multus/multus-588fk","openshift-multus/network-metrics-daemon-bvqvw","openshift-cluster-node-tuning-operator/tuned-r9cvk","openshift-monitoring/node-exporter-z8k8x","openshift-monitoring/prometheus-k8s-0","openshift-monitoring/thanos-querier-596fbdd49c-2nmrq","openshift-console/console-7698dd44-8ddgn","openshift-dns/node-resolver-tzgt2","openshift-image-registry/node-ca-77fcl","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-26.us-east-2.compute.internal","openshift-multus/multus-additional-cni-plugins-bltk8","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvhjt","openshift-insights/insights-runtime-extractor-r4spt","openshift-monitoring/prometheus-operator-9cd89b74c-6tbtd","openshift-network-operator/iptables-alerter-dg5c2"] Nov 01 19:59:15.315600 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.315579 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 01 19:59:15.315665 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.315656 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zgwx5\"" Nov 01 19:59:15.316160 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.316141 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-tdbxs\"" Nov 01 19:59:15.316160 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.316146 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-f8578\"" Nov 01 19:59:15.316304 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.316267 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 01 19:59:15.317712 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.317690 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 01 19:59:15.317887 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.317744 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 01 19:59:15.318067 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.317802 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 01 19:59:15.318174 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.318105 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 01 19:59:15.318267 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.318246 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 01 19:59:15.318357 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.318302 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 01 19:59:15.318451 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.318177 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rbpdw\"" Nov 01 19:59:15.318556 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.318538 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 01 19:59:15.318780 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.318758 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-jt8zj\"" Nov 01 19:59:15.318780 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.318774 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 01 19:59:15.319032 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.319003 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-n6s44\"" Nov 01 19:59:15.319140 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.319076 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 01 19:59:15.319227 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.319205 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-24gff\"" Nov 01 19:59:15.319298 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.319254 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 01 19:59:15.319298 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.319263 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 01 19:59:15.319526 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.319510 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 01 19:59:15.319638 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.319619 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-xcwxv\"" Nov 01 19:59:15.319719 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.319700 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 01 19:59:15.319823 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.319804 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 01 19:59:15.319873 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.319843 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-kjp59\"" Nov 01 19:59:15.319980 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.319962 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 01 19:59:15.320216 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.320200 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5qrts\"" Nov 01 19:59:15.320335 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.320210 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-j6fxj\"" Nov 01 19:59:15.320405 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.320369 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-754ws\"" Nov 01 19:59:15.320463 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.320420 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dgkqn\"" Nov 01 19:59:15.320511 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.320500 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 01 19:59:15.320709 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.320678 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 01 19:59:15.320811 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.320791 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 01 19:59:15.320866 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.320820 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 01 19:59:15.320866 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.320856 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 01 19:59:15.320956 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.320896 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 01 19:59:15.321062 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.321046 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 01 19:59:15.321111 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.321095 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 01 19:59:15.321184 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.321169 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 01 19:59:15.321258 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.321242 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 01 19:59:15.321369 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.321353 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 01 19:59:15.321413 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.321373 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 01 19:59:15.321464 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.321429 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 01 19:59:15.321565 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.320701 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 01 19:59:15.321840 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.321821 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 01 19:59:15.321930 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.321834 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 01 19:59:15.321930 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.321896 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 01 19:59:15.322045 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.321932 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9xfd7\"" Nov 01 19:59:15.322149 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.322127 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 01 19:59:15.322566 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.322550 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 01 19:59:15.322749 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.322635 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 01 19:59:15.322749 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.322641 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Nov 01 19:59:15.322749 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.322676 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 01 19:59:15.322749 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.322680 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 01 19:59:15.322749 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.322720 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 01 19:59:15.322749 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.322722 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-4cjlv\"" Nov 01 19:59:15.323033 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.322787 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 01 19:59:15.323033 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.322823 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-9pms2\"" Nov 01 19:59:15.323033 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.322839 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 01 19:59:15.323033 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.322844 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Nov 01 19:59:15.323033 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.322883 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 01 19:59:15.323367 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.323349 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a57adad-0c11-4902-be16-b8dfb4e3ee13-etc-kubernetes\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:59:15.323437 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.323378 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5a57adad-0c11-4902-be16-b8dfb4e3ee13-run\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:59:15.323437 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.323412 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2efc5bbf-efd4-4a42-bed8-53f2490bb7bc-original-pull-secret\") pod \"global-pull-secret-syncer-k5brd\" (UID: \"2efc5bbf-efd4-4a42-bed8-53f2490bb7bc\") " pod="kube-system/global-pull-secret-syncer-k5brd" Nov 01 19:59:15.323542 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.323485 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9e564268-fa85-48e6-8d1c-5069cc0f9648-cnibin\") pod \"multus-additional-cni-plugins-bltk8\" (UID: \"9e564268-fa85-48e6-8d1c-5069cc0f9648\") " pod="openshift-multus/multus-additional-cni-plugins-bltk8" Nov 01 19:59:15.323542 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.323517 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5a57adad-0c11-4902-be16-b8dfb4e3ee13-etc-sysconfig\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:59:15.323692 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.323542 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5a57adad-0c11-4902-be16-b8dfb4e3ee13-etc-sysctl-conf\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:59:15.323692 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.323641 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a57adad-0c11-4902-be16-b8dfb4e3ee13-var-lib-kubelet\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:59:15.323789 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.323698 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/91ef232e-99ae-42f2-be58-e17cc42f12ea-etc-selinux\") pod \"aws-ebs-csi-driver-node-fvhjt\" (UID: \"91ef232e-99ae-42f2-be58-e17cc42f12ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvhjt" Nov 01 19:59:15.323789 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.323744 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5a57adad-0c11-4902-be16-b8dfb4e3ee13-etc-sysctl-d\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:59:15.323893 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.323807 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5a57adad-0c11-4902-be16-b8dfb4e3ee13-etc-systemd\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:59:15.323893 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.323851 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2efc5bbf-efd4-4a42-bed8-53f2490bb7bc-original-pull-secret\") pod \"global-pull-secret-syncer-k5brd\" (UID: \"2efc5bbf-efd4-4a42-bed8-53f2490bb7bc\") " pod="kube-system/global-pull-secret-syncer-k5brd" Nov 01 19:59:15.323893 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.323854 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9e564268-fa85-48e6-8d1c-5069cc0f9648-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bltk8\" (UID: \"9e564268-fa85-48e6-8d1c-5069cc0f9648\") " pod="openshift-multus/multus-additional-cni-plugins-bltk8" Nov 01 19:59:15.324028 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.323914 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a57adad-0c11-4902-be16-b8dfb4e3ee13-tmp\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:59:15.324028 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.323949 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98mhk\" (UniqueName: \"kubernetes.io/projected/5a57adad-0c11-4902-be16-b8dfb4e3ee13-kube-api-access-98mhk\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:59:15.324028 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.324022 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/983a921d-61a5-4668-8e17-1a24b6eaf8b6-host\") pod \"node-ca-77fcl\" (UID: \"983a921d-61a5-4668-8e17-1a24b6eaf8b6\") " pod="openshift-image-registry/node-ca-77fcl" Nov 01 19:59:15.324171 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.324051 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fntm\" (UniqueName: \"kubernetes.io/projected/983a921d-61a5-4668-8e17-1a24b6eaf8b6-kube-api-access-8fntm\") pod \"node-ca-77fcl\" (UID: \"983a921d-61a5-4668-8e17-1a24b6eaf8b6\") " pod="openshift-image-registry/node-ca-77fcl" Nov 01 19:59:15.324171 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.324074 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/91ef232e-99ae-42f2-be58-e17cc42f12ea-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fvhjt\" (UID: \"91ef232e-99ae-42f2-be58-e17cc42f12ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvhjt" Nov 01 19:59:15.324171 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.324100 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/91ef232e-99ae-42f2-be58-e17cc42f12ea-socket-dir\") pod \"aws-ebs-csi-driver-node-fvhjt\" (UID: \"91ef232e-99ae-42f2-be58-e17cc42f12ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvhjt" Nov 01 19:59:15.324318 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.324176 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9e564268-fa85-48e6-8d1c-5069cc0f9648-system-cni-dir\") pod \"multus-additional-cni-plugins-bltk8\" (UID: \"9e564268-fa85-48e6-8d1c-5069cc0f9648\") " pod="openshift-multus/multus-additional-cni-plugins-bltk8" Nov 01 19:59:15.324318 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.324236 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5a57adad-0c11-4902-be16-b8dfb4e3ee13-tmp\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:59:15.324429 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.324407 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5a57adad-0c11-4902-be16-b8dfb4e3ee13-lib-modules\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:59:15.324502 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.324444 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2efc5bbf-efd4-4a42-bed8-53f2490bb7bc-kubelet-config\") pod \"global-pull-secret-syncer-k5brd\" (UID: \"2efc5bbf-efd4-4a42-bed8-53f2490bb7bc\") " pod="kube-system/global-pull-secret-syncer-k5brd" Nov 01 19:59:15.324502 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.324467 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9e564268-fa85-48e6-8d1c-5069cc0f9648-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bltk8\" (UID: \"9e564268-fa85-48e6-8d1c-5069cc0f9648\") " pod="openshift-multus/multus-additional-cni-plugins-bltk8" Nov 01 19:59:15.324502 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.324474 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9e564268-fa85-48e6-8d1c-5069cc0f9648-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bltk8\" (UID: \"9e564268-fa85-48e6-8d1c-5069cc0f9648\") " pod="openshift-multus/multus-additional-cni-plugins-bltk8" Nov 01 19:59:15.324502 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.324488 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a57adad-0c11-4902-be16-b8dfb4e3ee13-sys\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:59:15.324671 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.324535 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f16f57a8-9630-416c-bd95-f5a2c6af2f15-iptables-alerter-script\") pod \"iptables-alerter-dg5c2\" (UID: \"f16f57a8-9630-416c-bd95-f5a2c6af2f15\") " pod="openshift-network-operator/iptables-alerter-dg5c2" Nov 01 19:59:15.324671 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.324561 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f16f57a8-9630-416c-bd95-f5a2c6af2f15-host-slash\") pod \"iptables-alerter-dg5c2\" (UID: \"f16f57a8-9630-416c-bd95-f5a2c6af2f15\") " pod="openshift-network-operator/iptables-alerter-dg5c2" Nov 01 19:59:15.324671 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.324664 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/91ef232e-99ae-42f2-be58-e17cc42f12ea-device-dir\") pod \"aws-ebs-csi-driver-node-fvhjt\" (UID: \"91ef232e-99ae-42f2-be58-e17cc42f12ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvhjt" Nov 01 19:59:15.324835 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.324691 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/91ef232e-99ae-42f2-be58-e17cc42f12ea-sys-fs\") pod \"aws-ebs-csi-driver-node-fvhjt\" (UID: \"91ef232e-99ae-42f2-be58-e17cc42f12ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvhjt" Nov 01 19:59:15.324835 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.324731 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2efc5bbf-efd4-4a42-bed8-53f2490bb7bc-dbus\") pod \"global-pull-secret-syncer-k5brd\" (UID: \"2efc5bbf-efd4-4a42-bed8-53f2490bb7bc\") " pod="kube-system/global-pull-secret-syncer-k5brd" Nov 01 19:59:15.324835 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.324759 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9e564268-fa85-48e6-8d1c-5069cc0f9648-cni-binary-copy\") pod \"multus-additional-cni-plugins-bltk8\" (UID: \"9e564268-fa85-48e6-8d1c-5069cc0f9648\") " pod="openshift-multus/multus-additional-cni-plugins-bltk8" Nov 01 19:59:15.324835 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.324765 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f16f57a8-9630-416c-bd95-f5a2c6af2f15-iptables-alerter-script\") pod \"iptables-alerter-dg5c2\" (UID: \"f16f57a8-9630-416c-bd95-f5a2c6af2f15\") " pod="openshift-network-operator/iptables-alerter-dg5c2" Nov 01 19:59:15.324835 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.324784 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9e564268-fa85-48e6-8d1c-5069cc0f9648-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bltk8\" (UID: \"9e564268-fa85-48e6-8d1c-5069cc0f9648\") " pod="openshift-multus/multus-additional-cni-plugins-bltk8" Nov 01 19:59:15.325118 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.324846 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnjs4\" (UniqueName: \"kubernetes.io/projected/9e564268-fa85-48e6-8d1c-5069cc0f9648-kube-api-access-mnjs4\") pod \"multus-additional-cni-plugins-bltk8\" (UID: \"9e564268-fa85-48e6-8d1c-5069cc0f9648\") " pod="openshift-multus/multus-additional-cni-plugins-bltk8" Nov 01 19:59:15.325118 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.324882 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5a57adad-0c11-4902-be16-b8dfb4e3ee13-etc-modprobe-d\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:59:15.325118 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.324908 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/91ef232e-99ae-42f2-be58-e17cc42f12ea-registration-dir\") pod \"aws-ebs-csi-driver-node-fvhjt\" (UID: \"91ef232e-99ae-42f2-be58-e17cc42f12ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvhjt" Nov 01 19:59:15.325118 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.324993 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z6z7\" (UniqueName: \"kubernetes.io/projected/91ef232e-99ae-42f2-be58-e17cc42f12ea-kube-api-access-8z6z7\") pod \"aws-ebs-csi-driver-node-fvhjt\" (UID: \"91ef232e-99ae-42f2-be58-e17cc42f12ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvhjt" Nov 01 19:59:15.325118 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.325020 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a57adad-0c11-4902-be16-b8dfb4e3ee13-host\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:59:15.325118 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.325048 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5a57adad-0c11-4902-be16-b8dfb4e3ee13-etc-tuned\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:59:15.325118 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.325074 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/983a921d-61a5-4668-8e17-1a24b6eaf8b6-serviceca\") pod \"node-ca-77fcl\" (UID: \"983a921d-61a5-4668-8e17-1a24b6eaf8b6\") " pod="openshift-image-registry/node-ca-77fcl" Nov 01 19:59:15.325118 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.325100 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9xlj\" (UniqueName: \"kubernetes.io/projected/f16f57a8-9630-416c-bd95-f5a2c6af2f15-kube-api-access-j9xlj\") pod \"iptables-alerter-dg5c2\" (UID: \"f16f57a8-9630-416c-bd95-f5a2c6af2f15\") " pod="openshift-network-operator/iptables-alerter-dg5c2" Nov 01 19:59:15.325518 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.325306 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9e564268-fa85-48e6-8d1c-5069cc0f9648-cni-binary-copy\") pod \"multus-additional-cni-plugins-bltk8\" (UID: \"9e564268-fa85-48e6-8d1c-5069cc0f9648\") " pod="openshift-multus/multus-additional-cni-plugins-bltk8" Nov 01 19:59:15.325518 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.325477 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9e564268-fa85-48e6-8d1c-5069cc0f9648-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bltk8\" (UID: \"9e564268-fa85-48e6-8d1c-5069cc0f9648\") " pod="openshift-multus/multus-additional-cni-plugins-bltk8" Nov 01 19:59:15.325813 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.325792 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5a57adad-0c11-4902-be16-b8dfb4e3ee13-etc-tuned\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:59:15.326424 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.326402 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/983a921d-61a5-4668-8e17-1a24b6eaf8b6-serviceca\") pod \"node-ca-77fcl\" (UID: \"983a921d-61a5-4668-8e17-1a24b6eaf8b6\") " pod="openshift-image-registry/node-ca-77fcl" Nov 01 19:59:15.333814 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.333794 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fntm\" (UniqueName: \"kubernetes.io/projected/983a921d-61a5-4668-8e17-1a24b6eaf8b6-kube-api-access-8fntm\") pod \"node-ca-77fcl\" (UID: \"983a921d-61a5-4668-8e17-1a24b6eaf8b6\") " pod="openshift-image-registry/node-ca-77fcl" Nov 01 19:59:15.334069 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.334050 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98mhk\" (UniqueName: \"kubernetes.io/projected/5a57adad-0c11-4902-be16-b8dfb4e3ee13-kube-api-access-98mhk\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:59:15.334115 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.334056 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9xlj\" (UniqueName: \"kubernetes.io/projected/f16f57a8-9630-416c-bd95-f5a2c6af2f15-kube-api-access-j9xlj\") pod \"iptables-alerter-dg5c2\" (UID: \"f16f57a8-9630-416c-bd95-f5a2c6af2f15\") " pod="openshift-network-operator/iptables-alerter-dg5c2" Nov 01 19:59:15.334115 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.334095 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z6z7\" (UniqueName: \"kubernetes.io/projected/91ef232e-99ae-42f2-be58-e17cc42f12ea-kube-api-access-8z6z7\") pod \"aws-ebs-csi-driver-node-fvhjt\" (UID: \"91ef232e-99ae-42f2-be58-e17cc42f12ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvhjt" Nov 01 19:59:15.334267 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.334251 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnjs4\" (UniqueName: \"kubernetes.io/projected/9e564268-fa85-48e6-8d1c-5069cc0f9648-kube-api-access-mnjs4\") pod \"multus-additional-cni-plugins-bltk8\" (UID: \"9e564268-fa85-48e6-8d1c-5069cc0f9648\") " pod="openshift-multus/multus-additional-cni-plugins-bltk8" Nov 01 19:59:15.404475 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.404449 5935 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-26.us-east-2.compute.internal" Nov 01 19:59:15.404624 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.404604 5935 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-26.us-east-2.compute.internal" Nov 01 19:59:15.413844 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.413816 5935 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 01 19:59:15.415300 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.415268 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Nov 01 19:59:15.415402 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.415357 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 01 19:59:15.415603 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.415585 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 01 19:59:15.415682 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.415663 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Nov 01 19:59:15.415745 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.415675 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Nov 01 19:59:15.415805 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.415778 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Nov 01 19:59:15.415868 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.415802 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Nov 01 19:59:15.415868 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.415838 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 01 19:59:15.415938 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.415867 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Nov 01 19:59:15.415938 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.415913 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 01 19:59:15.416018 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.415968 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 01 19:59:15.416018 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.415967 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 01 19:59:15.416018 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.415982 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 01 19:59:15.416018 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.416001 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Nov 01 19:59:15.416159 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.416145 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 01 19:59:15.416226 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.416160 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 01 19:59:15.416226 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.416213 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Nov 01 19:59:15.417064 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.417046 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 01 19:59:15.417162 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.417066 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Nov 01 19:59:15.417162 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.417109 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Nov 01 19:59:15.417332 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.417322 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-dsvm4iuppcnu7\"" Nov 01 19:59:15.417919 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.417891 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 01 19:59:15.418018 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.417951 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 01 19:59:15.418018 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.417996 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Nov 01 19:59:15.418112 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.417919 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-ns62c\"" Nov 01 19:59:15.418252 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.418236 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-3vfuusupg38va\"" Nov 01 19:59:15.418420 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.418407 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 01 19:59:15.418672 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.418656 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 01 19:59:15.418921 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.418892 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 01 19:59:15.418996 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.418919 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-snwvx\"" Nov 01 19:59:15.421394 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.421374 5935 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 19:59:15.421495 ip-10-0-136-26 kubenswrapper[5935]: E1101 19:59:15.421440 5935 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-136-26.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-26.us-east-2.compute.internal" Nov 01 19:59:15.421920 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.421883 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-88k4l\"" Nov 01 19:59:15.422087 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.422068 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-j6j2k\"" Nov 01 19:59:15.422153 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.422089 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Nov 01 19:59:15.422153 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.422116 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 01 19:59:15.425693 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.425504 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/83a0642b-4dfe-4ddb-b506-f0654c7b6d30-crio-socket\") pod \"insights-runtime-extractor-r4spt\" (UID: \"83a0642b-4dfe-4ddb-b506-f0654c7b6d30\") " pod="openshift-insights/insights-runtime-extractor-r4spt" Nov 01 19:59:15.425693 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.425547 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq6fc\" (UniqueName: \"kubernetes.io/projected/40d57e7e-4896-4e0c-b266-ef5e6ca8beab-kube-api-access-nq6fc\") pod \"thanos-querier-596fbdd49c-2nmrq\" (UID: \"40d57e7e-4896-4e0c-b266-ef5e6ca8beab\") " pod="openshift-monitoring/thanos-querier-596fbdd49c-2nmrq" Nov 01 19:59:15.425693 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.425573 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a57adad-0c11-4902-be16-b8dfb4e3ee13-host\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:59:15.425693 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.425599 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-service-ca\") pod \"console-865bfb674b-jlr46\" (UID: \"87d7d5b7-5e96-4777-afcb-2a26d45b0ba8\") " pod="openshift-console/console-865bfb674b-jlr46" Nov 01 19:59:15.425693 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.425623 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/60e3a171-27e7-4ae6-ad6c-89984ecf5634-console-config\") pod \"console-7698dd44-8ddgn\" (UID: \"60e3a171-27e7-4ae6-ad6c-89984ecf5634\") " pod="openshift-console/console-7698dd44-8ddgn" Nov 01 19:59:15.425693 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.425647 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-host-var-lib-kubelet\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:59:15.425693 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.425671 5935 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 01 19:59:15.425693 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.425687 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5a57adad-0c11-4902-be16-b8dfb4e3ee13-host\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:59:15.426106 ip-10-0-136-26 kubenswrapper[5935]: E1101 19:59:15.425716 5935 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-136-26.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-26.us-east-2.compute.internal" Nov 01 19:59:15.426106 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.425737 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0e3a247-a792-431d-85cb-9b46982c3373-tmp-dir\") pod \"node-resolver-tzgt2\" (UID: \"c0e3a247-a792-431d-85cb-9b46982c3373\") " pod="openshift-dns/node-resolver-tzgt2" Nov 01 19:59:15.426106 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.425778 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/961c6886-e297-4202-b10a-df97f3537736-node-exporter-wtmp\") pod \"node-exporter-z8k8x\" (UID: \"961c6886-e297-4202-b10a-df97f3537736\") " pod="openshift-monitoring/node-exporter-z8k8x" Nov 01 19:59:15.426106 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.425804 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-host-cni-netd\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:59:15.426106 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.425832 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-ovnkube-script-lib\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:59:15.426106 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.425855 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65e6a200-c6f4-4671-9d2e-b0158ff3c5ce-metrics-certs\") pod \"network-metrics-daemon-bvqvw\" (UID: \"65e6a200-c6f4-4671-9d2e-b0158ff3c5ce\") " pod="openshift-multus/network-metrics-daemon-bvqvw" Nov 01 19:59:15.426106 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.426009 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-service-ca\") pod \"console-865bfb674b-jlr46\" (UID: \"87d7d5b7-5e96-4777-afcb-2a26d45b0ba8\") " pod="openshift-console/console-865bfb674b-jlr46" Nov 01 19:59:15.426546 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.426138 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/60e3a171-27e7-4ae6-ad6c-89984ecf5634-console-config\") pod \"console-7698dd44-8ddgn\" (UID: \"60e3a171-27e7-4ae6-ad6c-89984ecf5634\") " pod="openshift-console/console-7698dd44-8ddgn" Nov 01 19:59:15.426546 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.426223 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65e6a200-c6f4-4671-9d2e-b0158ff3c5ce-metrics-certs\") pod \"network-metrics-daemon-bvqvw\" (UID: \"65e6a200-c6f4-4671-9d2e-b0158ff3c5ce\") " pod="openshift-multus/network-metrics-daemon-bvqvw" Nov 01 19:59:15.426546 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.426264 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcwgs\" (UniqueName: \"kubernetes.io/projected/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-kube-api-access-pcwgs\") pod \"console-865bfb674b-jlr46\" (UID: \"87d7d5b7-5e96-4777-afcb-2a26d45b0ba8\") " pod="openshift-console/console-865bfb674b-jlr46" Nov 01 19:59:15.426546 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.426272 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-ovnkube-script-lib\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:59:15.426546 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.426311 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0e3a247-a792-431d-85cb-9b46982c3373-tmp-dir\") pod \"node-resolver-tzgt2\" (UID: \"c0e3a247-a792-431d-85cb-9b46982c3373\") " pod="openshift-dns/node-resolver-tzgt2" Nov 01 19:59:15.426546 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.426321 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/80c66ec7-5430-41a1-8641-56db5e4e9d73-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:15.426546 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.426349 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/60e3a171-27e7-4ae6-ad6c-89984ecf5634-console-oauth-config\") pod \"console-7698dd44-8ddgn\" (UID: \"60e3a171-27e7-4ae6-ad6c-89984ecf5634\") " pod="openshift-console/console-7698dd44-8ddgn" Nov 01 19:59:15.426546 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.426375 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fa82c276-5679-40f4-b7b4-e73eba7de9ef-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-6tbtd\" (UID: \"fa82c276-5679-40f4-b7b4-e73eba7de9ef\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6tbtd" Nov 01 19:59:15.426546 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.426399 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-oauth-serving-cert\") pod \"console-865bfb674b-jlr46\" (UID: \"87d7d5b7-5e96-4777-afcb-2a26d45b0ba8\") " pod="openshift-console/console-865bfb674b-jlr46" Nov 01 19:59:15.426546 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.426421 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:15.426546 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.426444 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/80c66ec7-5430-41a1-8641-56db5e4e9d73-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:15.426546 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.426502 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz4mx\" (UniqueName: \"kubernetes.io/projected/c0e3a247-a792-431d-85cb-9b46982c3373-kube-api-access-nz4mx\") pod \"node-resolver-tzgt2\" (UID: \"c0e3a247-a792-431d-85cb-9b46982c3373\") " pod="openshift-dns/node-resolver-tzgt2" Nov 01 19:59:15.426546 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.426543 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtnhd\" (UniqueName: \"kubernetes.io/projected/65e6a200-c6f4-4671-9d2e-b0158ff3c5ce-kube-api-access-jtnhd\") pod \"network-metrics-daemon-bvqvw\" (UID: \"65e6a200-c6f4-4671-9d2e-b0158ff3c5ce\") " pod="openshift-multus/network-metrics-daemon-bvqvw" Nov 01 19:59:15.427168 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.426569 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:15.427168 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.426595 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-host-run-ovn-kubernetes\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:59:15.427168 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.426619 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9e564268-fa85-48e6-8d1c-5069cc0f9648-system-cni-dir\") pod \"multus-additional-cni-plugins-bltk8\" (UID: \"9e564268-fa85-48e6-8d1c-5069cc0f9648\") " pod="openshift-multus/multus-additional-cni-plugins-bltk8" Nov 01 19:59:15.427168 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.426644 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-console-serving-cert\") pod \"console-865bfb674b-jlr46\" (UID: \"87d7d5b7-5e96-4777-afcb-2a26d45b0ba8\") " pod="openshift-console/console-865bfb674b-jlr46" Nov 01 19:59:15.427168 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.426669 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80c66ec7-5430-41a1-8641-56db5e4e9d73-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:15.427168 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.426694 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:59:15.427168 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.426718 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58167352-076f-40d7-abee-1c5f48f67566-tmp-dir\") pod \"dns-default-5lztx\" (UID: \"58167352-076f-40d7-abee-1c5f48f67566\") " pod="openshift-dns/dns-default-5lztx" Nov 01 19:59:15.427168 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.426743 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/40d57e7e-4896-4e0c-b266-ef5e6ca8beab-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-596fbdd49c-2nmrq\" (UID: \"40d57e7e-4896-4e0c-b266-ef5e6ca8beab\") " pod="openshift-monitoring/thanos-querier-596fbdd49c-2nmrq" Nov 01 19:59:15.427168 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.426774 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa82c276-5679-40f4-b7b4-e73eba7de9ef-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-6tbtd\" (UID: \"fa82c276-5679-40f4-b7b4-e73eba7de9ef\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6tbtd" Nov 01 19:59:15.427168 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.426808 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9e564268-fa85-48e6-8d1c-5069cc0f9648-system-cni-dir\") pod \"multus-additional-cni-plugins-bltk8\" (UID: \"9e564268-fa85-48e6-8d1c-5069cc0f9648\") " pod="openshift-multus/multus-additional-cni-plugins-bltk8" Nov 01 19:59:15.427168 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.426828 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/91ef232e-99ae-42f2-be58-e17cc42f12ea-registration-dir\") pod \"aws-ebs-csi-driver-node-fvhjt\" (UID: \"91ef232e-99ae-42f2-be58-e17cc42f12ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvhjt" Nov 01 19:59:15.427168 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.426852 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-console-config\") pod \"console-865bfb674b-jlr46\" (UID: \"87d7d5b7-5e96-4777-afcb-2a26d45b0ba8\") " pod="openshift-console/console-865bfb674b-jlr46" Nov 01 19:59:15.427168 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.426882 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq4k2\" (UniqueName: \"kubernetes.io/projected/16e51d04-90d5-484d-9f85-e9d484205220-kube-api-access-vq4k2\") pod \"network-check-target-b7mcz\" (UID: \"16e51d04-90d5-484d-9f85-e9d484205220\") " pod="openshift-network-diagnostics/network-check-target-b7mcz" Nov 01 19:59:15.427168 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.426906 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:15.427168 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.426931 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-run-openvswitch\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:59:15.427168 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.426954 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5a57adad-0c11-4902-be16-b8dfb4e3ee13-run\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:59:15.427168 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.426976 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5bdv\" (UniqueName: \"kubernetes.io/projected/961c6886-e297-4202-b10a-df97f3537736-kube-api-access-x5bdv\") pod \"node-exporter-z8k8x\" (UID: \"961c6886-e297-4202-b10a-df97f3537736\") " pod="openshift-monitoring/node-exporter-z8k8x" Nov 01 19:59:15.427168 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.426999 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/80c66ec7-5430-41a1-8641-56db5e4e9d73-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:15.427168 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.427053 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-web-config\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:15.427168 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.427078 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:15.427168 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.427101 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/58167352-076f-40d7-abee-1c5f48f67566-config-volume\") pod \"dns-default-5lztx\" (UID: \"58167352-076f-40d7-abee-1c5f48f67566\") " pod="openshift-dns/dns-default-5lztx" Nov 01 19:59:15.427168 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.427127 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-etc-kubernetes\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:59:15.427168 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.427152 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/961c6886-e297-4202-b10a-df97f3537736-node-exporter-tls\") pod \"node-exporter-z8k8x\" (UID: \"961c6886-e297-4202-b10a-df97f3537736\") " pod="openshift-monitoring/node-exporter-z8k8x" Nov 01 19:59:15.427168 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.427180 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/961c6886-e297-4202-b10a-df97f3537736-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-z8k8x\" (UID: \"961c6886-e297-4202-b10a-df97f3537736\") " pod="openshift-monitoring/node-exporter-z8k8x" Nov 01 19:59:15.428343 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.427210 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-host-cni-bin\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:59:15.428343 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.427246 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/83a0642b-4dfe-4ddb-b506-f0654c7b6d30-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r4spt\" (UID: \"83a0642b-4dfe-4ddb-b506-f0654c7b6d30\") " pod="openshift-insights/insights-runtime-extractor-r4spt" Nov 01 19:59:15.428343 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.427278 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/40d57e7e-4896-4e0c-b266-ef5e6ca8beab-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-596fbdd49c-2nmrq\" (UID: \"40d57e7e-4896-4e0c-b266-ef5e6ca8beab\") " pod="openshift-monitoring/thanos-querier-596fbdd49c-2nmrq" Nov 01 19:59:15.428343 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.427330 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/60e3a171-27e7-4ae6-ad6c-89984ecf5634-service-ca\") pod \"console-7698dd44-8ddgn\" (UID: \"60e3a171-27e7-4ae6-ad6c-89984ecf5634\") " pod="openshift-console/console-7698dd44-8ddgn" Nov 01 19:59:15.428343 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.427338 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fa82c276-5679-40f4-b7b4-e73eba7de9ef-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-6tbtd\" (UID: \"fa82c276-5679-40f4-b7b4-e73eba7de9ef\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6tbtd" Nov 01 19:59:15.428343 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.427369 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5a57adad-0c11-4902-be16-b8dfb4e3ee13-etc-sysctl-d\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:59:15.428343 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.427404 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5a57adad-0c11-4902-be16-b8dfb4e3ee13-etc-systemd\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:59:15.428343 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.427451 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-trusted-ca-bundle\") pod \"console-865bfb674b-jlr46\" (UID: \"87d7d5b7-5e96-4777-afcb-2a26d45b0ba8\") " pod="openshift-console/console-865bfb674b-jlr46" Nov 01 19:59:15.428343 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.427468 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:15.428343 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.427481 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80c66ec7-5430-41a1-8641-56db5e4e9d73-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:15.428343 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.427532 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-host-run-netns\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:59:15.428343 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.427607 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58167352-076f-40d7-abee-1c5f48f67566-tmp-dir\") pod \"dns-default-5lztx\" (UID: \"58167352-076f-40d7-abee-1c5f48f67566\") " pod="openshift-dns/dns-default-5lztx" Nov 01 19:59:15.428343 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.427607 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-oauth-serving-cert\") pod \"console-865bfb674b-jlr46\" (UID: \"87d7d5b7-5e96-4777-afcb-2a26d45b0ba8\") " pod="openshift-console/console-865bfb674b-jlr46" Nov 01 19:59:15.428343 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.427681 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/80c66ec7-5430-41a1-8641-56db5e4e9d73-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:15.428343 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.427781 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/40d57e7e-4896-4e0c-b266-ef5e6ca8beab-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-596fbdd49c-2nmrq\" (UID: \"40d57e7e-4896-4e0c-b266-ef5e6ca8beab\") " pod="openshift-monitoring/thanos-querier-596fbdd49c-2nmrq" Nov 01 19:59:15.428343 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.427855 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-etc-openvswitch\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:59:15.428343 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.427857 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80c66ec7-5430-41a1-8641-56db5e4e9d73-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:15.428343 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.427913 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:15.428343 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.427962 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/60e3a171-27e7-4ae6-ad6c-89984ecf5634-console-oauth-config\") pod \"console-7698dd44-8ddgn\" (UID: \"60e3a171-27e7-4ae6-ad6c-89984ecf5634\") " pod="openshift-console/console-7698dd44-8ddgn" Nov 01 19:59:15.428343 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.428190 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/60e3a171-27e7-4ae6-ad6c-89984ecf5634-service-ca\") pod \"console-7698dd44-8ddgn\" (UID: \"60e3a171-27e7-4ae6-ad6c-89984ecf5634\") " pod="openshift-console/console-7698dd44-8ddgn" Nov 01 19:59:15.428343 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.428250 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5a57adad-0c11-4902-be16-b8dfb4e3ee13-run\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:59:15.428343 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.428279 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/961c6886-e297-4202-b10a-df97f3537736-node-exporter-tls\") pod \"node-exporter-z8k8x\" (UID: \"961c6886-e297-4202-b10a-df97f3537736\") " pod="openshift-monitoring/node-exporter-z8k8x" Nov 01 19:59:15.429409 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.428407 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Nov 01 19:59:15.429409 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.428551 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa82c276-5679-40f4-b7b4-e73eba7de9ef-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-6tbtd\" (UID: \"fa82c276-5679-40f4-b7b4-e73eba7de9ef\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6tbtd" Nov 01 19:59:15.429409 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.428602 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-console-config\") pod \"console-865bfb674b-jlr46\" (UID: \"87d7d5b7-5e96-4777-afcb-2a26d45b0ba8\") " pod="openshift-console/console-865bfb674b-jlr46" Nov 01 19:59:15.429409 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.428661 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 01 19:59:15.429409 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.428710 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58167352-076f-40d7-abee-1c5f48f67566-metrics-tls\") pod \"dns-default-5lztx\" (UID: \"58167352-076f-40d7-abee-1c5f48f67566\") " pod="openshift-dns/dns-default-5lztx" Nov 01 19:59:15.429409 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.428713 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5a57adad-0c11-4902-be16-b8dfb4e3ee13-etc-sysctl-d\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:59:15.429409 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.428748 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/961c6886-e297-4202-b10a-df97f3537736-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-z8k8x\" (UID: \"961c6886-e297-4202-b10a-df97f3537736\") " pod="openshift-monitoring/node-exporter-z8k8x" Nov 01 19:59:15.429409 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.428767 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/91ef232e-99ae-42f2-be58-e17cc42f12ea-registration-dir\") pod \"aws-ebs-csi-driver-node-fvhjt\" (UID: \"91ef232e-99ae-42f2-be58-e17cc42f12ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvhjt" Nov 01 19:59:15.429409 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.428795 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-host-run-netns\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:59:15.429409 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.428838 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:15.429409 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.428866 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/40d57e7e-4896-4e0c-b266-ef5e6ca8beab-tmp\") pod \"thanos-querier-596fbdd49c-2nmrq\" (UID: \"40d57e7e-4896-4e0c-b266-ef5e6ca8beab\") " pod="openshift-monitoring/thanos-querier-596fbdd49c-2nmrq" Nov 01 19:59:15.429409 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.428892 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/961c6886-e297-4202-b10a-df97f3537736-node-exporter-accelerators-collector-config\") pod \"node-exporter-z8k8x\" (UID: \"961c6886-e297-4202-b10a-df97f3537736\") " pod="openshift-monitoring/node-exporter-z8k8x" Nov 01 19:59:15.429409 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.428922 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs2sl\" (UniqueName: \"kubernetes.io/projected/a225113f-cd49-48a9-b211-595bf5de763a-kube-api-access-fs2sl\") pod \"ingress-canary-tq7l5\" (UID: \"a225113f-cd49-48a9-b211-595bf5de763a\") " pod="openshift-ingress-canary/ingress-canary-tq7l5" Nov 01 19:59:15.429409 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.428930 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/58167352-076f-40d7-abee-1c5f48f67566-metrics-tls\") pod \"dns-default-5lztx\" (UID: \"58167352-076f-40d7-abee-1c5f48f67566\") " pod="openshift-dns/dns-default-5lztx" Nov 01 19:59:15.429409 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.428947 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-config\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:15.429409 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.428971 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5a57adad-0c11-4902-be16-b8dfb4e3ee13-etc-systemd\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:59:15.429409 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.428974 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-systemd-units\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:59:15.429409 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.429025 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80c66ec7-5430-41a1-8641-56db5e4e9d73-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:15.429409 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.429030 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9e564268-fa85-48e6-8d1c-5069cc0f9648-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bltk8\" (UID: \"9e564268-fa85-48e6-8d1c-5069cc0f9648\") " pod="openshift-multus/multus-additional-cni-plugins-bltk8" Nov 01 19:59:15.429409 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.429055 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a57adad-0c11-4902-be16-b8dfb4e3ee13-sys\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:59:15.429409 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.429071 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/80c66ec7-5430-41a1-8641-56db5e4e9d73-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:15.429409 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.429078 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/91ef232e-99ae-42f2-be58-e17cc42f12ea-device-dir\") pod \"aws-ebs-csi-driver-node-fvhjt\" (UID: \"91ef232e-99ae-42f2-be58-e17cc42f12ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvhjt" Nov 01 19:59:15.429409 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.429176 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/83a0642b-4dfe-4ddb-b506-f0654c7b6d30-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-r4spt\" (UID: \"83a0642b-4dfe-4ddb-b506-f0654c7b6d30\") " pod="openshift-insights/insights-runtime-extractor-r4spt" Nov 01 19:59:15.429409 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.429246 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:15.429409 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.429248 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/80c66ec7-5430-41a1-8641-56db5e4e9d73-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:15.429409 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.429278 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/80c66ec7-5430-41a1-8641-56db5e4e9d73-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:15.429409 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.429302 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5a57adad-0c11-4902-be16-b8dfb4e3ee13-sys\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:59:15.429409 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.429322 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-run-systemd\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:59:15.429409 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.429328 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/91ef232e-99ae-42f2-be58-e17cc42f12ea-device-dir\") pod \"aws-ebs-csi-driver-node-fvhjt\" (UID: \"91ef232e-99ae-42f2-be58-e17cc42f12ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvhjt" Nov 01 19:59:15.429409 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.429353 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/40d57e7e-4896-4e0c-b266-ef5e6ca8beab-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-596fbdd49c-2nmrq\" (UID: \"40d57e7e-4896-4e0c-b266-ef5e6ca8beab\") " pod="openshift-monitoring/thanos-querier-596fbdd49c-2nmrq" Nov 01 19:59:15.429409 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.429378 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/40d57e7e-4896-4e0c-b266-ef5e6ca8beab-tmp\") pod \"thanos-querier-596fbdd49c-2nmrq\" (UID: \"40d57e7e-4896-4e0c-b266-ef5e6ca8beab\") " pod="openshift-monitoring/thanos-querier-596fbdd49c-2nmrq" Nov 01 19:59:15.429409 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.429381 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5a57adad-0c11-4902-be16-b8dfb4e3ee13-etc-modprobe-d\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:59:15.429409 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.429409 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/62c9da55-b21d-4f8f-a3ec-98e823655351-multus-daemon-config\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:59:15.431558 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.429436 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/961c6886-e297-4202-b10a-df97f3537736-node-exporter-textfile\") pod \"node-exporter-z8k8x\" (UID: \"961c6886-e297-4202-b10a-df97f3537736\") " pod="openshift-monitoring/node-exporter-z8k8x" Nov 01 19:59:15.431558 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.429460 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a225113f-cd49-48a9-b211-595bf5de763a-cert\") pod \"ingress-canary-tq7l5\" (UID: \"a225113f-cd49-48a9-b211-595bf5de763a\") " pod="openshift-ingress-canary/ingress-canary-tq7l5" Nov 01 19:59:15.431558 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.429486 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a57adad-0c11-4902-be16-b8dfb4e3ee13-etc-kubernetes\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:59:15.431558 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.429510 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-multus-socket-dir-parent\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:59:15.431558 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.429521 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/961c6886-e297-4202-b10a-df97f3537736-node-exporter-accelerators-collector-config\") pod \"node-exporter-z8k8x\" (UID: \"961c6886-e297-4202-b10a-df97f3537736\") " pod="openshift-monitoring/node-exporter-z8k8x" Nov 01 19:59:15.431558 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.429539 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80c66ec7-5430-41a1-8641-56db5e4e9d73-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:15.431558 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.429568 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/80c66ec7-5430-41a1-8641-56db5e4e9d73-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:15.431558 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.429410 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 01 19:59:15.431558 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.429702 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9e564268-fa85-48e6-8d1c-5069cc0f9648-cnibin\") pod \"multus-additional-cni-plugins-bltk8\" (UID: \"9e564268-fa85-48e6-8d1c-5069cc0f9648\") " pod="openshift-multus/multus-additional-cni-plugins-bltk8" Nov 01 19:59:15.431558 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.429722 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5a57adad-0c11-4902-be16-b8dfb4e3ee13-etc-sysconfig\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:59:15.431558 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.429738 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/91ef232e-99ae-42f2-be58-e17cc42f12ea-etc-selinux\") pod \"aws-ebs-csi-driver-node-fvhjt\" (UID: \"91ef232e-99ae-42f2-be58-e17cc42f12ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvhjt" Nov 01 19:59:15.431558 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.429763 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7jcx\" (UniqueName: \"kubernetes.io/projected/fa82c276-5679-40f4-b7b4-e73eba7de9ef-kube-api-access-g7jcx\") pod \"prometheus-operator-9cd89b74c-6tbtd\" (UID: \"fa82c276-5679-40f4-b7b4-e73eba7de9ef\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6tbtd" Nov 01 19:59:15.431558 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.429781 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-console-oauth-config\") pod \"console-865bfb674b-jlr46\" (UID: \"87d7d5b7-5e96-4777-afcb-2a26d45b0ba8\") " pod="openshift-console/console-865bfb674b-jlr46" Nov 01 19:59:15.431558 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.429799 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/80c66ec7-5430-41a1-8641-56db5e4e9d73-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:15.431558 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.429815 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/83a0642b-4dfe-4ddb-b506-f0654c7b6d30-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r4spt\" (UID: \"83a0642b-4dfe-4ddb-b506-f0654c7b6d30\") " pod="openshift-insights/insights-runtime-extractor-r4spt" Nov 01 19:59:15.431558 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.429831 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:15.431558 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.429838 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm8bk\" (UniqueName: \"kubernetes.io/projected/83a0642b-4dfe-4ddb-b506-f0654c7b6d30-kube-api-access-bm8bk\") pod \"insights-runtime-extractor-r4spt\" (UID: \"83a0642b-4dfe-4ddb-b506-f0654c7b6d30\") " pod="openshift-insights/insights-runtime-extractor-r4spt" Nov 01 19:59:15.431558 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.429859 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8537e78c-5b2c-4966-9558-044ddc39119b-agent-certs\") pod \"konnectivity-agent-rz7h2\" (UID: \"8537e78c-5b2c-4966-9558-044ddc39119b\") " pod="kube-system/konnectivity-agent-rz7h2" Nov 01 19:59:15.431558 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.429875 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chtj4\" (UniqueName: \"kubernetes.io/projected/58167352-076f-40d7-abee-1c5f48f67566-kube-api-access-chtj4\") pod \"dns-default-5lztx\" (UID: \"58167352-076f-40d7-abee-1c5f48f67566\") " pod="openshift-dns/dns-default-5lztx" Nov 01 19:59:15.431558 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.429892 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/40d57e7e-4896-4e0c-b266-ef5e6ca8beab-secret-thanos-querier-tls\") pod \"thanos-querier-596fbdd49c-2nmrq\" (UID: \"40d57e7e-4896-4e0c-b266-ef5e6ca8beab\") " pod="openshift-monitoring/thanos-querier-596fbdd49c-2nmrq" Nov 01 19:59:15.431558 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.429921 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/40d57e7e-4896-4e0c-b266-ef5e6ca8beab-metrics-client-ca\") pod \"thanos-querier-596fbdd49c-2nmrq\" (UID: \"40d57e7e-4896-4e0c-b266-ef5e6ca8beab\") " pod="openshift-monitoring/thanos-querier-596fbdd49c-2nmrq" Nov 01 19:59:15.431558 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.429942 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/983a921d-61a5-4668-8e17-1a24b6eaf8b6-host\") pod \"node-ca-77fcl\" (UID: \"983a921d-61a5-4668-8e17-1a24b6eaf8b6\") " pod="openshift-image-registry/node-ca-77fcl" Nov 01 19:59:15.431558 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.429956 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/91ef232e-99ae-42f2-be58-e17cc42f12ea-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fvhjt\" (UID: \"91ef232e-99ae-42f2-be58-e17cc42f12ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvhjt" Nov 01 19:59:15.431558 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.429973 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-host-run-k8s-cni-cncf-io\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:59:15.431558 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.429993 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-run-ovn\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:59:15.431558 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.430014 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5a57adad-0c11-4902-be16-b8dfb4e3ee13-lib-modules\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:59:15.431558 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.430028 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/961c6886-e297-4202-b10a-df97f3537736-sys\") pod \"node-exporter-z8k8x\" (UID: \"961c6886-e297-4202-b10a-df97f3537736\") " pod="openshift-monitoring/node-exporter-z8k8x" Nov 01 19:59:15.431558 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.430046 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/961c6886-e297-4202-b10a-df97f3537736-metrics-client-ca\") pod \"node-exporter-z8k8x\" (UID: \"961c6886-e297-4202-b10a-df97f3537736\") " pod="openshift-monitoring/node-exporter-z8k8x" Nov 01 19:59:15.431558 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.430069 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:15.431558 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.430098 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80c66ec7-5430-41a1-8641-56db5e4e9d73-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:15.431558 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.430269 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:15.431558 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.430339 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/83a0642b-4dfe-4ddb-b506-f0654c7b6d30-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-r4spt\" (UID: \"83a0642b-4dfe-4ddb-b506-f0654c7b6d30\") " pod="openshift-insights/insights-runtime-extractor-r4spt" Nov 01 19:59:15.431558 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.430372 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-env-overrides\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:59:15.431558 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.430397 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/961c6886-e297-4202-b10a-df97f3537736-node-exporter-textfile\") pod \"node-exporter-z8k8x\" (UID: \"961c6886-e297-4202-b10a-df97f3537736\") " pod="openshift-monitoring/node-exporter-z8k8x" Nov 01 19:59:15.431558 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.430402 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/60e3a171-27e7-4ae6-ad6c-89984ecf5634-console-serving-cert\") pod \"console-7698dd44-8ddgn\" (UID: \"60e3a171-27e7-4ae6-ad6c-89984ecf5634\") " pod="openshift-console/console-7698dd44-8ddgn" Nov 01 19:59:15.431558 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.430436 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f16f57a8-9630-416c-bd95-f5a2c6af2f15-host-slash\") pod \"iptables-alerter-dg5c2\" (UID: \"f16f57a8-9630-416c-bd95-f5a2c6af2f15\") " pod="openshift-network-operator/iptables-alerter-dg5c2" Nov 01 19:59:15.431558 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.430465 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/91ef232e-99ae-42f2-be58-e17cc42f12ea-sys-fs\") pod \"aws-ebs-csi-driver-node-fvhjt\" (UID: \"91ef232e-99ae-42f2-be58-e17cc42f12ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvhjt" Nov 01 19:59:15.431558 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.430474 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-console-serving-cert\") pod \"console-865bfb674b-jlr46\" (UID: \"87d7d5b7-5e96-4777-afcb-2a26d45b0ba8\") " pod="openshift-console/console-865bfb674b-jlr46" Nov 01 19:59:15.431558 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.430494 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhv76\" (UniqueName: \"kubernetes.io/projected/62c9da55-b21d-4f8f-a3ec-98e823655351-kube-api-access-fhv76\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:59:15.431558 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.430524 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/fa82c276-5679-40f4-b7b4-e73eba7de9ef-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-6tbtd\" (UID: \"fa82c276-5679-40f4-b7b4-e73eba7de9ef\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6tbtd" Nov 01 19:59:15.431558 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.430541 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/80c66ec7-5430-41a1-8641-56db5e4e9d73-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:15.431558 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.430552 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4zhc\" (UniqueName: \"kubernetes.io/projected/80c66ec7-5430-41a1-8641-56db5e4e9d73-kube-api-access-x4zhc\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:15.431558 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.430588 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8537e78c-5b2c-4966-9558-044ddc39119b-agent-certs\") pod \"konnectivity-agent-rz7h2\" (UID: \"8537e78c-5b2c-4966-9558-044ddc39119b\") " pod="kube-system/konnectivity-agent-rz7h2" Nov 01 19:59:15.431558 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.430592 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f16f57a8-9630-416c-bd95-f5a2c6af2f15-host-slash\") pod \"iptables-alerter-dg5c2\" (UID: \"f16f57a8-9630-416c-bd95-f5a2c6af2f15\") " pod="openshift-network-operator/iptables-alerter-dg5c2" Nov 01 19:59:15.431558 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.430614 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/91ef232e-99ae-42f2-be58-e17cc42f12ea-sys-fs\") pod \"aws-ebs-csi-driver-node-fvhjt\" (UID: \"91ef232e-99ae-42f2-be58-e17cc42f12ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvhjt" Nov 01 19:59:15.431558 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.430619 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/40d57e7e-4896-4e0c-b266-ef5e6ca8beab-secret-thanos-querier-tls\") pod \"thanos-querier-596fbdd49c-2nmrq\" (UID: \"40d57e7e-4896-4e0c-b266-ef5e6ca8beab\") " pod="openshift-monitoring/thanos-querier-596fbdd49c-2nmrq" Nov 01 19:59:15.431558 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.430715 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/40d57e7e-4896-4e0c-b266-ef5e6ca8beab-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-596fbdd49c-2nmrq\" (UID: \"40d57e7e-4896-4e0c-b266-ef5e6ca8beab\") " pod="openshift-monitoring/thanos-querier-596fbdd49c-2nmrq" Nov 01 19:59:15.431558 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.430756 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a225113f-cd49-48a9-b211-595bf5de763a-cert\") pod \"ingress-canary-tq7l5\" (UID: \"a225113f-cd49-48a9-b211-595bf5de763a\") " pod="openshift-ingress-canary/ingress-canary-tq7l5" Nov 01 19:59:15.431558 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.430794 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5a57adad-0c11-4902-be16-b8dfb4e3ee13-etc-sysconfig\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:59:15.431558 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.430799 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5a57adad-0c11-4902-be16-b8dfb4e3ee13-etc-modprobe-d\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:59:15.431558 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.430823 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9e564268-fa85-48e6-8d1c-5069cc0f9648-cnibin\") pod \"multus-additional-cni-plugins-bltk8\" (UID: \"9e564268-fa85-48e6-8d1c-5069cc0f9648\") " pod="openshift-multus/multus-additional-cni-plugins-bltk8" Nov 01 19:59:15.431558 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.430821 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/fa82c276-5679-40f4-b7b4-e73eba7de9ef-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-6tbtd\" (UID: \"fa82c276-5679-40f4-b7b4-e73eba7de9ef\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6tbtd" Nov 01 19:59:15.431558 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.430840 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/91ef232e-99ae-42f2-be58-e17cc42f12ea-etc-selinux\") pod \"aws-ebs-csi-driver-node-fvhjt\" (UID: \"91ef232e-99ae-42f2-be58-e17cc42f12ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvhjt" Nov 01 19:59:15.431558 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.430857 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/91ef232e-99ae-42f2-be58-e17cc42f12ea-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fvhjt\" (UID: \"91ef232e-99ae-42f2-be58-e17cc42f12ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvhjt" Nov 01 19:59:15.431558 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.430900 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/80c66ec7-5430-41a1-8641-56db5e4e9d73-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:15.431558 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.430958 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-env-overrides\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:59:15.431558 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.430973 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:15.431558 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.431020 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/40d57e7e-4896-4e0c-b266-ef5e6ca8beab-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-596fbdd49c-2nmrq\" (UID: \"40d57e7e-4896-4e0c-b266-ef5e6ca8beab\") " pod="openshift-monitoring/thanos-querier-596fbdd49c-2nmrq" Nov 01 19:59:15.431558 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.431039 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/60e3a171-27e7-4ae6-ad6c-89984ecf5634-console-serving-cert\") pod \"console-7698dd44-8ddgn\" (UID: \"60e3a171-27e7-4ae6-ad6c-89984ecf5634\") " pod="openshift-console/console-7698dd44-8ddgn" Nov 01 19:59:15.431558 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.431101 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-config\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:15.431558 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.431127 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-console-oauth-config\") pod \"console-865bfb674b-jlr46\" (UID: \"87d7d5b7-5e96-4777-afcb-2a26d45b0ba8\") " pod="openshift-console/console-865bfb674b-jlr46" Nov 01 19:59:15.431558 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.431205 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-trusted-ca-bundle\") pod \"console-865bfb674b-jlr46\" (UID: \"87d7d5b7-5e96-4777-afcb-2a26d45b0ba8\") " pod="openshift-console/console-865bfb674b-jlr46" Nov 01 19:59:15.431558 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.431220 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/62c9da55-b21d-4f8f-a3ec-98e823655351-multus-daemon-config\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:59:15.431558 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.431259 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/983a921d-61a5-4668-8e17-1a24b6eaf8b6-host\") pod \"node-ca-77fcl\" (UID: \"983a921d-61a5-4668-8e17-1a24b6eaf8b6\") " pod="openshift-image-registry/node-ca-77fcl" Nov 01 19:59:15.431558 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.431266 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a57adad-0c11-4902-be16-b8dfb4e3ee13-etc-kubernetes\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:59:15.431558 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.431427 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9e564268-fa85-48e6-8d1c-5069cc0f9648-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bltk8\" (UID: \"9e564268-fa85-48e6-8d1c-5069cc0f9648\") " pod="openshift-multus/multus-additional-cni-plugins-bltk8" Nov 01 19:59:15.431558 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.431514 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5a57adad-0c11-4902-be16-b8dfb4e3ee13-lib-modules\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:59:15.431558 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.431521 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/961c6886-e297-4202-b10a-df97f3537736-metrics-client-ca\") pod \"node-exporter-z8k8x\" (UID: \"961c6886-e297-4202-b10a-df97f3537736\") " pod="openshift-monitoring/node-exporter-z8k8x" Nov 01 19:59:15.431558 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.431538 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-system-cni-dir\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:59:15.431558 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.431558 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-host-kubelet\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:59:15.431558 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.431576 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-var-lib-openvswitch\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:59:15.434826 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.431606 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/40d57e7e-4896-4e0c-b266-ef5e6ca8beab-secret-grpc-tls\") pod \"thanos-querier-596fbdd49c-2nmrq\" (UID: \"40d57e7e-4896-4e0c-b266-ef5e6ca8beab\") " pod="openshift-monitoring/thanos-querier-596fbdd49c-2nmrq" Nov 01 19:59:15.434826 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.431624 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-multus-conf-dir\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:59:15.434826 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.431640 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c0e3a247-a792-431d-85cb-9b46982c3373-hosts-file\") pod \"node-resolver-tzgt2\" (UID: \"c0e3a247-a792-431d-85cb-9b46982c3373\") " pod="openshift-dns/node-resolver-tzgt2" Nov 01 19:59:15.434826 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.431656 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/961c6886-e297-4202-b10a-df97f3537736-root\") pod \"node-exporter-z8k8x\" (UID: \"961c6886-e297-4202-b10a-df97f3537736\") " pod="openshift-monitoring/node-exporter-z8k8x" Nov 01 19:59:15.434826 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.431672 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5a57adad-0c11-4902-be16-b8dfb4e3ee13-etc-sysctl-conf\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:59:15.434826 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.431680 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/58167352-076f-40d7-abee-1c5f48f67566-config-volume\") pod \"dns-default-5lztx\" (UID: \"58167352-076f-40d7-abee-1c5f48f67566\") " pod="openshift-dns/dns-default-5lztx" Nov 01 19:59:15.434826 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.431689 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a57adad-0c11-4902-be16-b8dfb4e3ee13-var-lib-kubelet\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:59:15.434826 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.431709 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-hostroot\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:59:15.434826 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.431953 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-web-config\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:15.434826 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.432242 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/40d57e7e-4896-4e0c-b266-ef5e6ca8beab-metrics-client-ca\") pod \"thanos-querier-596fbdd49c-2nmrq\" (UID: \"40d57e7e-4896-4e0c-b266-ef5e6ca8beab\") " pod="openshift-monitoring/thanos-querier-596fbdd49c-2nmrq" Nov 01 19:59:15.434826 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.432582 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/40d57e7e-4896-4e0c-b266-ef5e6ca8beab-secret-grpc-tls\") pod \"thanos-querier-596fbdd49c-2nmrq\" (UID: \"40d57e7e-4896-4e0c-b266-ef5e6ca8beab\") " pod="openshift-monitoring/thanos-querier-596fbdd49c-2nmrq" Nov 01 19:59:15.434826 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.432721 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/80c66ec7-5430-41a1-8641-56db5e4e9d73-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:15.434826 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.432840 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtnhd\" (UniqueName: \"kubernetes.io/projected/65e6a200-c6f4-4671-9d2e-b0158ff3c5ce-kube-api-access-jtnhd\") pod \"network-metrics-daemon-bvqvw\" (UID: \"65e6a200-c6f4-4671-9d2e-b0158ff3c5ce\") " pod="openshift-multus/network-metrics-daemon-bvqvw" Nov 01 19:59:15.434826 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.432849 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-host-slash\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:59:15.434826 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.432893 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60e3a171-27e7-4ae6-ad6c-89984ecf5634-trusted-ca-bundle\") pod \"console-7698dd44-8ddgn\" (UID: \"60e3a171-27e7-4ae6-ad6c-89984ecf5634\") " pod="openshift-console/console-7698dd44-8ddgn" Nov 01 19:59:15.434826 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.432923 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-cnibin\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:59:15.434826 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.432951 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-host-var-lib-cni-bin\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:59:15.434826 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.432975 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-host-run-multus-certs\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:59:15.434826 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.433004 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/80c66ec7-5430-41a1-8641-56db5e4e9d73-config-out\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:15.434826 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.433032 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-node-log\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:59:15.434826 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.433059 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq6fc\" (UniqueName: \"kubernetes.io/projected/40d57e7e-4896-4e0c-b266-ef5e6ca8beab-kube-api-access-nq6fc\") pod \"thanos-querier-596fbdd49c-2nmrq\" (UID: \"40d57e7e-4896-4e0c-b266-ef5e6ca8beab\") " pod="openshift-monitoring/thanos-querier-596fbdd49c-2nmrq" Nov 01 19:59:15.434826 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.433140 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-log-socket\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:59:15.434826 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.433171 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-ovnkube-config\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:59:15.434826 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.433202 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/40d57e7e-4896-4e0c-b266-ef5e6ca8beab-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-596fbdd49c-2nmrq\" (UID: \"40d57e7e-4896-4e0c-b266-ef5e6ca8beab\") " pod="openshift-monitoring/thanos-querier-596fbdd49c-2nmrq" Nov 01 19:59:15.434826 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.433237 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/91ef232e-99ae-42f2-be58-e17cc42f12ea-socket-dir\") pod \"aws-ebs-csi-driver-node-fvhjt\" (UID: \"91ef232e-99ae-42f2-be58-e17cc42f12ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvhjt" Nov 01 19:59:15.434826 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.433267 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2efc5bbf-efd4-4a42-bed8-53f2490bb7bc-kubelet-config\") pod \"global-pull-secret-syncer-k5brd\" (UID: \"2efc5bbf-efd4-4a42-bed8-53f2490bb7bc\") " pod="kube-system/global-pull-secret-syncer-k5brd" Nov 01 19:59:15.434826 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.433318 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-multus-cni-dir\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:59:15.434826 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.433340 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz4mx\" (UniqueName: \"kubernetes.io/projected/c0e3a247-a792-431d-85cb-9b46982c3373-kube-api-access-nz4mx\") pod \"node-resolver-tzgt2\" (UID: \"c0e3a247-a792-431d-85cb-9b46982c3373\") " pod="openshift-dns/node-resolver-tzgt2" Nov 01 19:59:15.434826 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.433348 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/62c9da55-b21d-4f8f-a3ec-98e823655351-cni-binary-copy\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:59:15.434826 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.433376 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:15.434826 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.433411 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-ovn-node-metrics-cert\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:59:15.434826 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.433438 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-host-var-lib-cni-multus\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:59:15.434826 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.433439 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/80c66ec7-5430-41a1-8641-56db5e4e9d73-config-out\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:15.434826 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.433461 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5a57adad-0c11-4902-be16-b8dfb4e3ee13-etc-sysctl-conf\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:59:15.434826 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.433501 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dg6b\" (UniqueName: \"kubernetes.io/projected/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-kube-api-access-6dg6b\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:59:15.434826 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.433527 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/91ef232e-99ae-42f2-be58-e17cc42f12ea-socket-dir\") pod \"aws-ebs-csi-driver-node-fvhjt\" (UID: \"91ef232e-99ae-42f2-be58-e17cc42f12ea\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fvhjt" Nov 01 19:59:15.434826 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.433536 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-ovnkube-config\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:59:15.434826 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.433565 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a57adad-0c11-4902-be16-b8dfb4e3ee13-var-lib-kubelet\") pod \"tuned-r9cvk\" (UID: \"5a57adad-0c11-4902-be16-b8dfb4e3ee13\") " pod="openshift-cluster-node-tuning-operator/tuned-r9cvk" Nov 01 19:59:15.434826 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.433662 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/40d57e7e-4896-4e0c-b266-ef5e6ca8beab-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-596fbdd49c-2nmrq\" (UID: \"40d57e7e-4896-4e0c-b266-ef5e6ca8beab\") " pod="openshift-monitoring/thanos-querier-596fbdd49c-2nmrq" Nov 01 19:59:15.434826 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.433713 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2efc5bbf-efd4-4a42-bed8-53f2490bb7bc-dbus\") pod \"global-pull-secret-syncer-k5brd\" (UID: \"2efc5bbf-efd4-4a42-bed8-53f2490bb7bc\") " pod="kube-system/global-pull-secret-syncer-k5brd" Nov 01 19:59:15.434826 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.433837 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2efc5bbf-efd4-4a42-bed8-53f2490bb7bc-kubelet-config\") pod \"global-pull-secret-syncer-k5brd\" (UID: \"2efc5bbf-efd4-4a42-bed8-53f2490bb7bc\") " pod="kube-system/global-pull-secret-syncer-k5brd" Nov 01 19:59:15.434826 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.433895 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/62c9da55-b21d-4f8f-a3ec-98e823655351-cni-binary-copy\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:59:15.434826 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.433877 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:15.434826 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.433940 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcwgs\" (UniqueName: \"kubernetes.io/projected/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-kube-api-access-pcwgs\") pod \"console-865bfb674b-jlr46\" (UID: \"87d7d5b7-5e96-4777-afcb-2a26d45b0ba8\") " pod="openshift-console/console-865bfb674b-jlr46" Nov 01 19:59:15.434826 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.433964 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-tmp\") pod \"console-865bfb674b-jlr46\" (UID: \"87d7d5b7-5e96-4777-afcb-2a26d45b0ba8\") " pod="openshift-console/console-865bfb674b-jlr46" Nov 01 19:59:15.434826 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.433997 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2efc5bbf-efd4-4a42-bed8-53f2490bb7bc-dbus\") pod \"global-pull-secret-syncer-k5brd\" (UID: \"2efc5bbf-efd4-4a42-bed8-53f2490bb7bc\") " pod="kube-system/global-pull-secret-syncer-k5brd" Nov 01 19:59:15.434826 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.434005 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8537e78c-5b2c-4966-9558-044ddc39119b-konnectivity-ca\") pod \"konnectivity-agent-rz7h2\" (UID: \"8537e78c-5b2c-4966-9558-044ddc39119b\") " pod="kube-system/konnectivity-agent-rz7h2" Nov 01 19:59:15.434826 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.434111 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-tmp\") pod \"console-865bfb674b-jlr46\" (UID: \"87d7d5b7-5e96-4777-afcb-2a26d45b0ba8\") " pod="openshift-console/console-865bfb674b-jlr46" Nov 01 19:59:15.434826 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.434147 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/83a0642b-4dfe-4ddb-b506-f0654c7b6d30-data-volume\") pod \"insights-runtime-extractor-r4spt\" (UID: \"83a0642b-4dfe-4ddb-b506-f0654c7b6d30\") " pod="openshift-insights/insights-runtime-extractor-r4spt" Nov 01 19:59:15.434826 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.434174 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/60e3a171-27e7-4ae6-ad6c-89984ecf5634-tmp\") pod \"console-7698dd44-8ddgn\" (UID: \"60e3a171-27e7-4ae6-ad6c-89984ecf5634\") " pod="openshift-console/console-7698dd44-8ddgn" Nov 01 19:59:15.434826 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.434238 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/60e3a171-27e7-4ae6-ad6c-89984ecf5634-oauth-serving-cert\") pod \"console-7698dd44-8ddgn\" (UID: \"60e3a171-27e7-4ae6-ad6c-89984ecf5634\") " pod="openshift-console/console-7698dd44-8ddgn" Nov 01 19:59:15.434826 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.434272 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cq6h\" (UniqueName: \"kubernetes.io/projected/60e3a171-27e7-4ae6-ad6c-89984ecf5634-kube-api-access-4cq6h\") pod \"console-7698dd44-8ddgn\" (UID: \"60e3a171-27e7-4ae6-ad6c-89984ecf5634\") " pod="openshift-console/console-7698dd44-8ddgn" Nov 01 19:59:15.434826 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.434380 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/83a0642b-4dfe-4ddb-b506-f0654c7b6d30-data-volume\") pod \"insights-runtime-extractor-r4spt\" (UID: \"83a0642b-4dfe-4ddb-b506-f0654c7b6d30\") " pod="openshift-insights/insights-runtime-extractor-r4spt" Nov 01 19:59:15.434826 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.434389 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60e3a171-27e7-4ae6-ad6c-89984ecf5634-trusted-ca-bundle\") pod \"console-7698dd44-8ddgn\" (UID: \"60e3a171-27e7-4ae6-ad6c-89984ecf5634\") " pod="openshift-console/console-7698dd44-8ddgn" Nov 01 19:59:15.434826 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.434237 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-ovn-node-metrics-cert\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:59:15.434826 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.434463 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/60e3a171-27e7-4ae6-ad6c-89984ecf5634-tmp\") pod \"console-7698dd44-8ddgn\" (UID: \"60e3a171-27e7-4ae6-ad6c-89984ecf5634\") " pod="openshift-console/console-7698dd44-8ddgn" Nov 01 19:59:15.434826 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.434536 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5bdv\" (UniqueName: \"kubernetes.io/projected/961c6886-e297-4202-b10a-df97f3537736-kube-api-access-x5bdv\") pod \"node-exporter-z8k8x\" (UID: \"961c6886-e297-4202-b10a-df97f3537736\") " pod="openshift-monitoring/node-exporter-z8k8x" Nov 01 19:59:15.434826 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.434556 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8537e78c-5b2c-4966-9558-044ddc39119b-konnectivity-ca\") pod \"konnectivity-agent-rz7h2\" (UID: \"8537e78c-5b2c-4966-9558-044ddc39119b\") " pod="kube-system/konnectivity-agent-rz7h2" Nov 01 19:59:15.434826 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.434824 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/60e3a171-27e7-4ae6-ad6c-89984ecf5634-oauth-serving-cert\") pod \"console-7698dd44-8ddgn\" (UID: \"60e3a171-27e7-4ae6-ad6c-89984ecf5634\") " pod="openshift-console/console-7698dd44-8ddgn" Nov 01 19:59:15.436481 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.434894 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq4k2\" (UniqueName: \"kubernetes.io/projected/16e51d04-90d5-484d-9f85-e9d484205220-kube-api-access-vq4k2\") pod \"network-check-target-b7mcz\" (UID: \"16e51d04-90d5-484d-9f85-e9d484205220\") " pod="openshift-network-diagnostics/network-check-target-b7mcz" Nov 01 19:59:15.436481 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.435888 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs2sl\" (UniqueName: \"kubernetes.io/projected/a225113f-cd49-48a9-b211-595bf5de763a-kube-api-access-fs2sl\") pod \"ingress-canary-tq7l5\" (UID: \"a225113f-cd49-48a9-b211-595bf5de763a\") " pod="openshift-ingress-canary/ingress-canary-tq7l5" Nov 01 19:59:15.440321 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.440302 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chtj4\" (UniqueName: \"kubernetes.io/projected/58167352-076f-40d7-abee-1c5f48f67566-kube-api-access-chtj4\") pod \"dns-default-5lztx\" (UID: \"58167352-076f-40d7-abee-1c5f48f67566\") " pod="openshift-dns/dns-default-5lztx" Nov 01 19:59:15.440673 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.440652 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm8bk\" (UniqueName: \"kubernetes.io/projected/83a0642b-4dfe-4ddb-b506-f0654c7b6d30-kube-api-access-bm8bk\") pod \"insights-runtime-extractor-r4spt\" (UID: \"83a0642b-4dfe-4ddb-b506-f0654c7b6d30\") " pod="openshift-insights/insights-runtime-extractor-r4spt" Nov 01 19:59:15.441217 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.441198 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhv76\" (UniqueName: \"kubernetes.io/projected/62c9da55-b21d-4f8f-a3ec-98e823655351-kube-api-access-fhv76\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:59:15.441385 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.441368 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7jcx\" (UniqueName: \"kubernetes.io/projected/fa82c276-5679-40f4-b7b4-e73eba7de9ef-kube-api-access-g7jcx\") pod \"prometheus-operator-9cd89b74c-6tbtd\" (UID: \"fa82c276-5679-40f4-b7b4-e73eba7de9ef\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-6tbtd" Nov 01 19:59:15.442966 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.442946 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cq6h\" (UniqueName: \"kubernetes.io/projected/60e3a171-27e7-4ae6-ad6c-89984ecf5634-kube-api-access-4cq6h\") pod \"console-7698dd44-8ddgn\" (UID: \"60e3a171-27e7-4ae6-ad6c-89984ecf5634\") " pod="openshift-console/console-7698dd44-8ddgn" Nov 01 19:59:15.443106 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.443011 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dg6b\" (UniqueName: \"kubernetes.io/projected/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-kube-api-access-6dg6b\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:59:15.443200 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.443179 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4zhc\" (UniqueName: \"kubernetes.io/projected/80c66ec7-5430-41a1-8641-56db5e4e9d73-kube-api-access-x4zhc\") pod \"prometheus-k8s-0\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:15.535698 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.535666 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-host-kubelet\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:59:15.535698 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.535708 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-var-lib-openvswitch\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:59:15.535865 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.535735 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-multus-conf-dir\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:59:15.535865 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.535769 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c0e3a247-a792-431d-85cb-9b46982c3373-hosts-file\") pod \"node-resolver-tzgt2\" (UID: \"c0e3a247-a792-431d-85cb-9b46982c3373\") " pod="openshift-dns/node-resolver-tzgt2" Nov 01 19:59:15.535865 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.535774 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-host-kubelet\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:59:15.535865 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.535791 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/961c6886-e297-4202-b10a-df97f3537736-root\") pod \"node-exporter-z8k8x\" (UID: \"961c6886-e297-4202-b10a-df97f3537736\") " pod="openshift-monitoring/node-exporter-z8k8x" Nov 01 19:59:15.535865 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.535842 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-var-lib-openvswitch\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:59:15.535865 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.535840 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-multus-conf-dir\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:59:15.536060 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.535864 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c0e3a247-a792-431d-85cb-9b46982c3373-hosts-file\") pod \"node-resolver-tzgt2\" (UID: \"c0e3a247-a792-431d-85cb-9b46982c3373\") " pod="openshift-dns/node-resolver-tzgt2" Nov 01 19:59:15.536060 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.535893 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-hostroot\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:59:15.536060 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.535920 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-host-slash\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:59:15.536060 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.535923 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/961c6886-e297-4202-b10a-df97f3537736-root\") pod \"node-exporter-z8k8x\" (UID: \"961c6886-e297-4202-b10a-df97f3537736\") " pod="openshift-monitoring/node-exporter-z8k8x" Nov 01 19:59:15.536060 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.535940 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-cnibin\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:59:15.536060 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.535943 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-hostroot\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:59:15.536060 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.535957 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-host-var-lib-cni-bin\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:59:15.536060 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.535969 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-host-slash\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:59:15.536060 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.535986 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-cnibin\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:59:15.536060 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.535972 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-host-run-multus-certs\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:59:15.536060 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.536003 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-host-var-lib-cni-bin\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:59:15.536060 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.536008 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-host-run-multus-certs\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:59:15.536060 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.536022 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-node-log\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:59:15.536060 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.536040 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-log-socket\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:59:15.536060 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.536059 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-multus-cni-dir\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:59:15.536628 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.536077 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-host-var-lib-cni-multus\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:59:15.536628 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.536096 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-log-socket\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:59:15.536628 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.536104 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/83a0642b-4dfe-4ddb-b506-f0654c7b6d30-crio-socket\") pod \"insights-runtime-extractor-r4spt\" (UID: \"83a0642b-4dfe-4ddb-b506-f0654c7b6d30\") " pod="openshift-insights/insights-runtime-extractor-r4spt" Nov 01 19:59:15.536628 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.536122 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-node-log\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:59:15.536628 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.536135 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-host-var-lib-cni-multus\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:59:15.536628 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.536135 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-host-var-lib-kubelet\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:59:15.536628 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.536152 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-multus-cni-dir\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:59:15.536628 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.536171 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/83a0642b-4dfe-4ddb-b506-f0654c7b6d30-crio-socket\") pod \"insights-runtime-extractor-r4spt\" (UID: \"83a0642b-4dfe-4ddb-b506-f0654c7b6d30\") " pod="openshift-insights/insights-runtime-extractor-r4spt" Nov 01 19:59:15.536628 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.536176 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-host-var-lib-kubelet\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:59:15.536628 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.536172 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/961c6886-e297-4202-b10a-df97f3537736-node-exporter-wtmp\") pod \"node-exporter-z8k8x\" (UID: \"961c6886-e297-4202-b10a-df97f3537736\") " pod="openshift-monitoring/node-exporter-z8k8x" Nov 01 19:59:15.536628 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.536216 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/961c6886-e297-4202-b10a-df97f3537736-node-exporter-wtmp\") pod \"node-exporter-z8k8x\" (UID: \"961c6886-e297-4202-b10a-df97f3537736\") " pod="openshift-monitoring/node-exporter-z8k8x" Nov 01 19:59:15.536628 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.536217 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-host-cni-netd\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:59:15.536628 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.536245 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-host-cni-netd\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:59:15.536628 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.536262 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-host-run-ovn-kubernetes\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:59:15.536628 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.536295 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:59:15.536628 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.536316 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-host-run-ovn-kubernetes\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:59:15.536628 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.536320 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-run-openvswitch\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:59:15.536628 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.536342 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-etc-kubernetes\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:59:15.536628 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.536359 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-host-cni-bin\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:59:15.536628 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.536361 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-run-openvswitch\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:59:15.536628 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.536343 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:59:15.536628 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.536377 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-host-run-netns\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:59:15.536628 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.536405 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-etc-openvswitch\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:59:15.536628 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.536408 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-host-cni-bin\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:59:15.536628 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.536404 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-host-run-netns\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:59:15.536628 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.536404 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-etc-kubernetes\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:59:15.536628 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.536430 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-host-run-netns\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:59:15.536628 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.536440 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-etc-openvswitch\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:59:15.536628 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.536450 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-systemd-units\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:59:15.536628 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.536479 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-systemd-units\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:59:15.536628 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.536484 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-host-run-netns\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:59:15.536628 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.536488 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-run-systemd\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:59:15.536628 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.536523 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-run-systemd\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:59:15.536628 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.536526 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-multus-socket-dir-parent\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:59:15.536628 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.536570 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-host-run-k8s-cni-cncf-io\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:59:15.536628 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.536597 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-run-ovn\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:59:15.536628 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.536602 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-host-run-k8s-cni-cncf-io\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:59:15.536628 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.536573 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-multus-socket-dir-parent\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:59:15.536628 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.536622 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/961c6886-e297-4202-b10a-df97f3537736-sys\") pod \"node-exporter-z8k8x\" (UID: \"961c6886-e297-4202-b10a-df97f3537736\") " pod="openshift-monitoring/node-exporter-z8k8x" Nov 01 19:59:15.536628 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.536643 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1cd0e9b2-272f-46f0-88d3-89f58c5dab49-run-ovn\") pod \"ovnkube-node-cffbl\" (UID: \"1cd0e9b2-272f-46f0-88d3-89f58c5dab49\") " pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:59:15.537687 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.536653 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-system-cni-dir\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:59:15.537687 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.536676 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/961c6886-e297-4202-b10a-df97f3537736-sys\") pod \"node-exporter-z8k8x\" (UID: \"961c6886-e297-4202-b10a-df97f3537736\") " pod="openshift-monitoring/node-exporter-z8k8x" Nov 01 19:59:15.537687 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:15.536728 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/62c9da55-b21d-4f8f-a3ec-98e823655351-system-cni-dir\") pod \"multus-588fk\" (UID: \"62c9da55-b21d-4f8f-a3ec-98e823655351\") " pod="openshift-multus/multus-588fk" Nov 01 19:59:16.496213 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:16.496187 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5lztx" Nov 01 19:59:17.368584 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:17.368553 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:17.383404 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:17.383381 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:18.224488 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:18.224451 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-596fbdd49c-2nmrq" Nov 01 19:59:18.543217 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:18.543152 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:59:22.937191 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:22.937162 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:59:22.937688 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:22.937382 5935 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 01 19:59:23.107850 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:23.107822 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7698dd44-8ddgn" Nov 01 19:59:23.588981 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:23.588947 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-b7mcz" Nov 01 19:59:25.599399 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:25.599319 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7698dd44-8ddgn" Nov 01 19:59:30.745423 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:30.745381 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cffbl" Nov 01 19:59:33.113105 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:33.113077 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7698dd44-8ddgn" Nov 01 19:59:33.116803 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:33.116782 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7698dd44-8ddgn" Nov 01 19:59:33.234446 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:33.234415 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5fccf54c48-xhbhp"] Nov 01 19:59:33.251484 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:33.251457 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fccf54c48-xhbhp" Nov 01 19:59:33.252115 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:33.252096 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5fccf54c48-xhbhp"] Nov 01 19:59:33.373769 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:33.373708 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d871b82e-7d5c-4faf-8918-74cf2b465c82-console-config\") pod \"console-5fccf54c48-xhbhp\" (UID: \"d871b82e-7d5c-4faf-8918-74cf2b465c82\") " pod="openshift-console/console-5fccf54c48-xhbhp" Nov 01 19:59:33.373769 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:33.373741 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d871b82e-7d5c-4faf-8918-74cf2b465c82-console-oauth-config\") pod \"console-5fccf54c48-xhbhp\" (UID: \"d871b82e-7d5c-4faf-8918-74cf2b465c82\") " pod="openshift-console/console-5fccf54c48-xhbhp" Nov 01 19:59:33.373769 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:33.373767 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d871b82e-7d5c-4faf-8918-74cf2b465c82-tmp\") pod \"console-5fccf54c48-xhbhp\" (UID: \"d871b82e-7d5c-4faf-8918-74cf2b465c82\") " pod="openshift-console/console-5fccf54c48-xhbhp" Nov 01 19:59:33.373964 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:33.373861 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d871b82e-7d5c-4faf-8918-74cf2b465c82-oauth-serving-cert\") pod \"console-5fccf54c48-xhbhp\" (UID: \"d871b82e-7d5c-4faf-8918-74cf2b465c82\") " pod="openshift-console/console-5fccf54c48-xhbhp" Nov 01 19:59:33.373964 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:33.373898 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d871b82e-7d5c-4faf-8918-74cf2b465c82-console-serving-cert\") pod \"console-5fccf54c48-xhbhp\" (UID: \"d871b82e-7d5c-4faf-8918-74cf2b465c82\") " pod="openshift-console/console-5fccf54c48-xhbhp" Nov 01 19:59:33.373964 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:33.373914 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhw5h\" (UniqueName: \"kubernetes.io/projected/d871b82e-7d5c-4faf-8918-74cf2b465c82-kube-api-access-fhw5h\") pod \"console-5fccf54c48-xhbhp\" (UID: \"d871b82e-7d5c-4faf-8918-74cf2b465c82\") " pod="openshift-console/console-5fccf54c48-xhbhp" Nov 01 19:59:33.373964 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:33.373955 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d871b82e-7d5c-4faf-8918-74cf2b465c82-service-ca\") pod \"console-5fccf54c48-xhbhp\" (UID: \"d871b82e-7d5c-4faf-8918-74cf2b465c82\") " pod="openshift-console/console-5fccf54c48-xhbhp" Nov 01 19:59:33.374085 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:33.374017 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d871b82e-7d5c-4faf-8918-74cf2b465c82-trusted-ca-bundle\") pod \"console-5fccf54c48-xhbhp\" (UID: \"d871b82e-7d5c-4faf-8918-74cf2b465c82\") " pod="openshift-console/console-5fccf54c48-xhbhp" Nov 01 19:59:33.474702 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:33.474665 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d871b82e-7d5c-4faf-8918-74cf2b465c82-tmp\") pod \"console-5fccf54c48-xhbhp\" (UID: \"d871b82e-7d5c-4faf-8918-74cf2b465c82\") " pod="openshift-console/console-5fccf54c48-xhbhp" Nov 01 19:59:33.474702 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:33.474709 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d871b82e-7d5c-4faf-8918-74cf2b465c82-oauth-serving-cert\") pod \"console-5fccf54c48-xhbhp\" (UID: \"d871b82e-7d5c-4faf-8918-74cf2b465c82\") " pod="openshift-console/console-5fccf54c48-xhbhp" Nov 01 19:59:33.474881 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:33.474737 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d871b82e-7d5c-4faf-8918-74cf2b465c82-console-serving-cert\") pod \"console-5fccf54c48-xhbhp\" (UID: \"d871b82e-7d5c-4faf-8918-74cf2b465c82\") " pod="openshift-console/console-5fccf54c48-xhbhp" Nov 01 19:59:33.474881 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:33.474761 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhw5h\" (UniqueName: \"kubernetes.io/projected/d871b82e-7d5c-4faf-8918-74cf2b465c82-kube-api-access-fhw5h\") pod \"console-5fccf54c48-xhbhp\" (UID: \"d871b82e-7d5c-4faf-8918-74cf2b465c82\") " pod="openshift-console/console-5fccf54c48-xhbhp" Nov 01 19:59:33.474881 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:33.474802 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d871b82e-7d5c-4faf-8918-74cf2b465c82-service-ca\") pod \"console-5fccf54c48-xhbhp\" (UID: \"d871b82e-7d5c-4faf-8918-74cf2b465c82\") " pod="openshift-console/console-5fccf54c48-xhbhp" Nov 01 19:59:33.474881 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:33.474838 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d871b82e-7d5c-4faf-8918-74cf2b465c82-trusted-ca-bundle\") pod \"console-5fccf54c48-xhbhp\" (UID: \"d871b82e-7d5c-4faf-8918-74cf2b465c82\") " pod="openshift-console/console-5fccf54c48-xhbhp" Nov 01 19:59:33.475069 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:33.474969 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d871b82e-7d5c-4faf-8918-74cf2b465c82-console-config\") pod \"console-5fccf54c48-xhbhp\" (UID: \"d871b82e-7d5c-4faf-8918-74cf2b465c82\") " pod="openshift-console/console-5fccf54c48-xhbhp" Nov 01 19:59:33.475069 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:33.475026 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d871b82e-7d5c-4faf-8918-74cf2b465c82-console-oauth-config\") pod \"console-5fccf54c48-xhbhp\" (UID: \"d871b82e-7d5c-4faf-8918-74cf2b465c82\") " pod="openshift-console/console-5fccf54c48-xhbhp" Nov 01 19:59:33.475171 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:33.475073 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d871b82e-7d5c-4faf-8918-74cf2b465c82-tmp\") pod \"console-5fccf54c48-xhbhp\" (UID: \"d871b82e-7d5c-4faf-8918-74cf2b465c82\") " pod="openshift-console/console-5fccf54c48-xhbhp" Nov 01 19:59:33.475245 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:33.475161 5935 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 01 19:59:33.475839 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:33.475808 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d871b82e-7d5c-4faf-8918-74cf2b465c82-trusted-ca-bundle\") pod \"console-5fccf54c48-xhbhp\" (UID: \"d871b82e-7d5c-4faf-8918-74cf2b465c82\") " pod="openshift-console/console-5fccf54c48-xhbhp" Nov 01 19:59:33.475950 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:33.475911 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d871b82e-7d5c-4faf-8918-74cf2b465c82-console-config\") pod \"console-5fccf54c48-xhbhp\" (UID: \"d871b82e-7d5c-4faf-8918-74cf2b465c82\") " pod="openshift-console/console-5fccf54c48-xhbhp" Nov 01 19:59:33.476013 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:33.475944 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d871b82e-7d5c-4faf-8918-74cf2b465c82-service-ca\") pod \"console-5fccf54c48-xhbhp\" (UID: \"d871b82e-7d5c-4faf-8918-74cf2b465c82\") " pod="openshift-console/console-5fccf54c48-xhbhp" Nov 01 19:59:33.476071 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:33.476004 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d871b82e-7d5c-4faf-8918-74cf2b465c82-oauth-serving-cert\") pod \"console-5fccf54c48-xhbhp\" (UID: \"d871b82e-7d5c-4faf-8918-74cf2b465c82\") " pod="openshift-console/console-5fccf54c48-xhbhp" Nov 01 19:59:33.477394 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:33.477371 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d871b82e-7d5c-4faf-8918-74cf2b465c82-console-serving-cert\") pod \"console-5fccf54c48-xhbhp\" (UID: \"d871b82e-7d5c-4faf-8918-74cf2b465c82\") " pod="openshift-console/console-5fccf54c48-xhbhp" Nov 01 19:59:33.477658 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:33.477643 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d871b82e-7d5c-4faf-8918-74cf2b465c82-console-oauth-config\") pod \"console-5fccf54c48-xhbhp\" (UID: \"d871b82e-7d5c-4faf-8918-74cf2b465c82\") " pod="openshift-console/console-5fccf54c48-xhbhp" Nov 01 19:59:33.482113 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:33.482090 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhw5h\" (UniqueName: \"kubernetes.io/projected/d871b82e-7d5c-4faf-8918-74cf2b465c82-kube-api-access-fhw5h\") pod \"console-5fccf54c48-xhbhp\" (UID: \"d871b82e-7d5c-4faf-8918-74cf2b465c82\") " pod="openshift-console/console-5fccf54c48-xhbhp" Nov 01 19:59:33.574058 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:33.574034 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fccf54c48-xhbhp" Nov 01 19:59:33.874738 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:33.874705 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5fccf54c48-xhbhp"] Nov 01 19:59:33.878928 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:33.878877 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd871b82e_7d5c_4faf_8918_74cf2b465c82.slice/crio-9bc3fd11e891a838e51fc87d17150cf0b641d6c1df4b059ab01a87736e29aea5 WatchSource:0}: Error finding container 9bc3fd11e891a838e51fc87d17150cf0b641d6c1df4b059ab01a87736e29aea5: Status 404 returned error can't find the container with id 9bc3fd11e891a838e51fc87d17150cf0b641d6c1df4b059ab01a87736e29aea5 Nov 01 19:59:34.472010 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:34.471978 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fccf54c48-xhbhp" event={"ID":"d871b82e-7d5c-4faf-8918-74cf2b465c82","Type":"ContainerStarted","Data":"c155f2808224aabae92d5bcbdc177cbd544aa0acfb558119404484a9d9dd1ac8"} Nov 01 19:59:34.472410 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:34.472034 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fccf54c48-xhbhp" event={"ID":"d871b82e-7d5c-4faf-8918-74cf2b465c82","Type":"ContainerStarted","Data":"9bc3fd11e891a838e51fc87d17150cf0b641d6c1df4b059ab01a87736e29aea5"} Nov 01 19:59:34.489993 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:34.489950 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5fccf54c48-xhbhp" podStartSLOduration=1.489930881 podStartE2EDuration="1.489930881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 19:59:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:59:34.48888114 +0000 UTC m=+20.262041652" watchObservedRunningTime="2026-11-01 19:59:34.489930881 +0000 UTC m=+20.263091387" Nov 01 19:59:34.959050 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:34.959024 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-rz7h2" Nov 01 19:59:35.366469 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:35.366379 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 19:59:35.366759 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:35.366716 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="80c66ec7-5430-41a1-8641-56db5e4e9d73" containerName="kube-rbac-proxy" containerID="cri-o://24713a72c5a0df1f0334a2d435e98bdb64cedb75a2385deb57868ffbb5c8f878" gracePeriod=600 Nov 01 19:59:35.366759 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:35.366716 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="80c66ec7-5430-41a1-8641-56db5e4e9d73" containerName="kube-rbac-proxy-web" containerID="cri-o://517de8fdee23b80434e71946162632465d7c9580a7b69ade708ffdf401cdd144" gracePeriod=600 Nov 01 19:59:35.366909 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:35.366808 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="80c66ec7-5430-41a1-8641-56db5e4e9d73" containerName="thanos-sidecar" containerID="cri-o://e25643c77686e3395636e0b4a10f7904a73836d08aeba79db39bfad411bb3f0d" gracePeriod=600 Nov 01 19:59:35.366965 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:35.366917 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="80c66ec7-5430-41a1-8641-56db5e4e9d73" containerName="config-reloader" containerID="cri-o://39ca1779d3e2325f0cc084bff04614b589b1f8e0c119fd669272dbd3c28dd02c" gracePeriod=600 Nov 01 19:59:35.367084 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:35.367062 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="80c66ec7-5430-41a1-8641-56db5e4e9d73" containerName="prometheus" containerID="cri-o://c51258636e3bb432293783857dddfce6b61d33076f8dcaf32df54d4d9da2f10e" gracePeriod=600 Nov 01 19:59:35.367187 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:35.367169 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="80c66ec7-5430-41a1-8641-56db5e4e9d73" containerName="kube-rbac-proxy-thanos" containerID="cri-o://54513e5b3e839c8bdba0581043ed5eac858babb463bc57d6f83f597d437a96a2" gracePeriod=600 Nov 01 19:59:35.646808 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:35.646733 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:35.842715 ip-10-0-136-26 kubenswrapper[5935]: E1101 19:59:35.842682 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 19:59:36.485758 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:36.485724 5935 generic.go:356] "Generic (PLEG): container finished" podID="80c66ec7-5430-41a1-8641-56db5e4e9d73" containerID="54513e5b3e839c8bdba0581043ed5eac858babb463bc57d6f83f597d437a96a2" exitCode=0 Nov 01 19:59:36.485758 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:36.485754 5935 generic.go:356] "Generic (PLEG): container finished" podID="80c66ec7-5430-41a1-8641-56db5e4e9d73" containerID="24713a72c5a0df1f0334a2d435e98bdb64cedb75a2385deb57868ffbb5c8f878" exitCode=0 Nov 01 19:59:36.485758 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:36.485764 5935 generic.go:356] "Generic (PLEG): container finished" podID="80c66ec7-5430-41a1-8641-56db5e4e9d73" containerID="517de8fdee23b80434e71946162632465d7c9580a7b69ade708ffdf401cdd144" exitCode=0 Nov 01 19:59:36.486003 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:36.485774 5935 generic.go:356] "Generic (PLEG): container finished" podID="80c66ec7-5430-41a1-8641-56db5e4e9d73" containerID="e25643c77686e3395636e0b4a10f7904a73836d08aeba79db39bfad411bb3f0d" exitCode=0 Nov 01 19:59:36.486003 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:36.485786 5935 generic.go:356] "Generic (PLEG): container finished" podID="80c66ec7-5430-41a1-8641-56db5e4e9d73" containerID="39ca1779d3e2325f0cc084bff04614b589b1f8e0c119fd669272dbd3c28dd02c" exitCode=0 Nov 01 19:59:36.486003 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:36.485794 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"80c66ec7-5430-41a1-8641-56db5e4e9d73","Type":"ContainerDied","Data":"54513e5b3e839c8bdba0581043ed5eac858babb463bc57d6f83f597d437a96a2"} Nov 01 19:59:36.486003 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:36.485854 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"80c66ec7-5430-41a1-8641-56db5e4e9d73","Type":"ContainerDied","Data":"24713a72c5a0df1f0334a2d435e98bdb64cedb75a2385deb57868ffbb5c8f878"} Nov 01 19:59:36.486003 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:36.485884 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"80c66ec7-5430-41a1-8641-56db5e4e9d73","Type":"ContainerDied","Data":"517de8fdee23b80434e71946162632465d7c9580a7b69ade708ffdf401cdd144"} Nov 01 19:59:36.486003 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:36.485907 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"80c66ec7-5430-41a1-8641-56db5e4e9d73","Type":"ContainerDied","Data":"e25643c77686e3395636e0b4a10f7904a73836d08aeba79db39bfad411bb3f0d"} Nov 01 19:59:36.486003 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:36.485932 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"80c66ec7-5430-41a1-8641-56db5e4e9d73","Type":"ContainerDied","Data":"39ca1779d3e2325f0cc084bff04614b589b1f8e0c119fd669272dbd3c28dd02c"} Nov 01 19:59:40.340707 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:40.340664 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-865bfb674b-jlr46" podUID="87d7d5b7-5e96-4777-afcb-2a26d45b0ba8" containerName="console" containerID="cri-o://e55af4d50013937a201d4ede50c7a2b53a2e9849997d886c823d63c8e96f0130" gracePeriod=15 Nov 01 19:59:40.508052 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:40.507970 5935 generic.go:356] "Generic (PLEG): container finished" podID="87d7d5b7-5e96-4777-afcb-2a26d45b0ba8" containerID="e55af4d50013937a201d4ede50c7a2b53a2e9849997d886c823d63c8e96f0130" exitCode=0 Nov 01 19:59:40.508155 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:40.508061 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-865bfb674b-jlr46" event={"ID":"87d7d5b7-5e96-4777-afcb-2a26d45b0ba8","Type":"ContainerDied","Data":"e55af4d50013937a201d4ede50c7a2b53a2e9849997d886c823d63c8e96f0130"} Nov 01 19:59:40.576557 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:40.576533 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-865bfb674b-jlr46" Nov 01 19:59:40.736857 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:40.736827 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-console-serving-cert\") pod \"87d7d5b7-5e96-4777-afcb-2a26d45b0ba8\" (UID: \"87d7d5b7-5e96-4777-afcb-2a26d45b0ba8\") " Nov 01 19:59:40.737025 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:40.736862 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-console-config\" (UniqueName: \"kubernetes.io/configmap/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-console-config\") pod \"87d7d5b7-5e96-4777-afcb-2a26d45b0ba8\" (UID: \"87d7d5b7-5e96-4777-afcb-2a26d45b0ba8\") " Nov 01 19:59:40.737025 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:40.736898 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-kube-api-access-pcwgs\" (UniqueName: \"kubernetes.io/projected/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-kube-api-access-pcwgs\") pod \"87d7d5b7-5e96-4777-afcb-2a26d45b0ba8\" (UID: \"87d7d5b7-5e96-4777-afcb-2a26d45b0ba8\") " Nov 01 19:59:40.737025 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:40.736927 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-oauth-serving-cert\") pod \"87d7d5b7-5e96-4777-afcb-2a26d45b0ba8\" (UID: \"87d7d5b7-5e96-4777-afcb-2a26d45b0ba8\") " Nov 01 19:59:40.737025 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:40.736959 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-trusted-ca-bundle\") pod \"87d7d5b7-5e96-4777-afcb-2a26d45b0ba8\" (UID: \"87d7d5b7-5e96-4777-afcb-2a26d45b0ba8\") " Nov 01 19:59:40.737025 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:40.737018 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-tmp\" (UniqueName: \"kubernetes.io/empty-dir/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-tmp\") pod \"87d7d5b7-5e96-4777-afcb-2a26d45b0ba8\" (UID: \"87d7d5b7-5e96-4777-afcb-2a26d45b0ba8\") " Nov 01 19:59:40.737263 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:40.737093 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-console-oauth-config\") pod \"87d7d5b7-5e96-4777-afcb-2a26d45b0ba8\" (UID: \"87d7d5b7-5e96-4777-afcb-2a26d45b0ba8\") " Nov 01 19:59:40.737263 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:40.737133 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-service-ca\" (UniqueName: \"kubernetes.io/configmap/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-service-ca\") pod \"87d7d5b7-5e96-4777-afcb-2a26d45b0ba8\" (UID: \"87d7d5b7-5e96-4777-afcb-2a26d45b0ba8\") " Nov 01 19:59:40.737447 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:40.737375 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-tmp" pod "87d7d5b7-5e96-4777-afcb-2a26d45b0ba8" (UID: "87d7d5b7-5e96-4777-afcb-2a26d45b0ba8"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 19:59:40.737447 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:40.737402 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-console-config" pod "87d7d5b7-5e96-4777-afcb-2a26d45b0ba8" (UID: "87d7d5b7-5e96-4777-afcb-2a26d45b0ba8"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 19:59:40.737903 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:40.737692 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-service-ca" pod "87d7d5b7-5e96-4777-afcb-2a26d45b0ba8" (UID: "87d7d5b7-5e96-4777-afcb-2a26d45b0ba8"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 19:59:40.737903 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:40.737747 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-oauth-serving-cert" pod "87d7d5b7-5e96-4777-afcb-2a26d45b0ba8" (UID: "87d7d5b7-5e96-4777-afcb-2a26d45b0ba8"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 19:59:40.737903 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:40.737767 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-trusted-ca-bundle" pod "87d7d5b7-5e96-4777-afcb-2a26d45b0ba8" (UID: "87d7d5b7-5e96-4777-afcb-2a26d45b0ba8"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 19:59:40.739872 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:40.739845 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-console-serving-cert" pod "87d7d5b7-5e96-4777-afcb-2a26d45b0ba8" (UID: "87d7d5b7-5e96-4777-afcb-2a26d45b0ba8"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 19:59:40.739983 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:40.739891 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-kube-api-access-pcwgs" pod "87d7d5b7-5e96-4777-afcb-2a26d45b0ba8" (UID: "87d7d5b7-5e96-4777-afcb-2a26d45b0ba8"). InnerVolumeSpecName "kube-api-access-pcwgs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 19:59:40.739983 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:40.739936 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-console-oauth-config" pod "87d7d5b7-5e96-4777-afcb-2a26d45b0ba8" (UID: "87d7d5b7-5e96-4777-afcb-2a26d45b0ba8"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 19:59:40.838236 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:40.838208 5935 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-tmp\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 19:59:40.838236 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:40.838237 5935 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-console-oauth-config\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 19:59:40.838384 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:40.838251 5935 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-service-ca\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 19:59:40.838384 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:40.838264 5935 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-console-serving-cert\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 19:59:40.838384 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:40.838277 5935 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-console-config\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 19:59:40.838384 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:40.838304 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcwgs\" (UniqueName: \"kubernetes.io/projected/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-kube-api-access-pcwgs\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 19:59:40.838384 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:40.838318 5935 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-oauth-serving-cert\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 19:59:40.838384 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:40.838331 5935 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8-trusted-ca-bundle\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 19:59:41.513638 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:41.513595 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-865bfb674b-jlr46" event={"ID":"87d7d5b7-5e96-4777-afcb-2a26d45b0ba8","Type":"ContainerDied","Data":"e943b81be985021134f7b01fa18e9de2d046185a015ab296c031b2318a819812"} Nov 01 19:59:41.513638 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:41.513615 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-865bfb674b-jlr46" Nov 01 19:59:41.514196 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:41.513661 5935 scope.go:122] "RemoveContainer" containerID="e55af4d50013937a201d4ede50c7a2b53a2e9849997d886c823d63c8e96f0130" Nov 01 19:59:41.533574 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:41.533553 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-865bfb674b-jlr46"] Nov 01 19:59:41.537418 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:41.537398 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-865bfb674b-jlr46"] Nov 01 19:59:41.898985 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:41.898962 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:42.046422 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.046337 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"80c66ec7-5430-41a1-8641-56db5e4e9d73\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " Nov 01 19:59:42.046422 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.046380 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80c66ec7-5430-41a1-8641-56db5e4e9d73-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80c66ec7-5430-41a1-8641-56db5e4e9d73-prometheus-trusted-ca-bundle\") pod \"80c66ec7-5430-41a1-8641-56db5e4e9d73\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " Nov 01 19:59:42.046422 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.046419 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-secret-prometheus-k8s-tls\") pod \"80c66ec7-5430-41a1-8641-56db5e4e9d73\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " Nov 01 19:59:42.046711 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.046448 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80c66ec7-5430-41a1-8641-56db5e4e9d73-config-out\" (UniqueName: \"kubernetes.io/empty-dir/80c66ec7-5430-41a1-8641-56db5e4e9d73-config-out\") pod \"80c66ec7-5430-41a1-8641-56db5e4e9d73\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " Nov 01 19:59:42.046711 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.046476 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"80c66ec7-5430-41a1-8641-56db5e4e9d73\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " Nov 01 19:59:42.046711 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.046508 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-web-config\" (UniqueName: \"kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-web-config\") pod \"80c66ec7-5430-41a1-8641-56db5e4e9d73\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " Nov 01 19:59:42.046711 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.046534 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-config\" (UniqueName: \"kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-config\") pod \"80c66ec7-5430-41a1-8641-56db5e4e9d73\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " Nov 01 19:59:42.046711 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.046568 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80c66ec7-5430-41a1-8641-56db5e4e9d73-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/80c66ec7-5430-41a1-8641-56db5e4e9d73-prometheus-k8s-rulefiles-0\") pod \"80c66ec7-5430-41a1-8641-56db5e4e9d73\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " Nov 01 19:59:42.046711 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.046601 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-secret-metrics-client-certs\") pod \"80c66ec7-5430-41a1-8641-56db5e4e9d73\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " Nov 01 19:59:42.046711 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.046629 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80c66ec7-5430-41a1-8641-56db5e4e9d73-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/80c66ec7-5430-41a1-8641-56db5e4e9d73-prometheus-k8s-db\") pod \"80c66ec7-5430-41a1-8641-56db5e4e9d73\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " Nov 01 19:59:42.046711 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.046665 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80c66ec7-5430-41a1-8641-56db5e4e9d73-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80c66ec7-5430-41a1-8641-56db5e4e9d73-configmap-kubelet-serving-ca-bundle\") pod \"80c66ec7-5430-41a1-8641-56db5e4e9d73\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " Nov 01 19:59:42.046711 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.046701 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80c66ec7-5430-41a1-8641-56db5e4e9d73-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/80c66ec7-5430-41a1-8641-56db5e4e9d73-configmap-metrics-client-ca\") pod \"80c66ec7-5430-41a1-8641-56db5e4e9d73\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " Nov 01 19:59:42.047158 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.046734 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-thanos-prometheus-http-client-file\") pod \"80c66ec7-5430-41a1-8641-56db5e4e9d73\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " Nov 01 19:59:42.047158 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.046795 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80c66ec7-5430-41a1-8641-56db5e4e9d73-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/80c66ec7-5430-41a1-8641-56db5e4e9d73-prometheus-k8s-rulefiles-2\") pod \"80c66ec7-5430-41a1-8641-56db5e4e9d73\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " Nov 01 19:59:42.047158 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.046830 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80c66ec7-5430-41a1-8641-56db5e4e9d73-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80c66ec7-5430-41a1-8641-56db5e4e9d73-configmap-serving-certs-ca-bundle\") pod \"80c66ec7-5430-41a1-8641-56db5e4e9d73\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " Nov 01 19:59:42.047158 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.046859 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-secret-kube-rbac-proxy\") pod \"80c66ec7-5430-41a1-8641-56db5e4e9d73\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " Nov 01 19:59:42.047158 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.046909 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-secret-grpc-tls\") pod \"80c66ec7-5430-41a1-8641-56db5e4e9d73\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " Nov 01 19:59:42.047158 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.046948 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80c66ec7-5430-41a1-8641-56db5e4e9d73-kube-api-access-x4zhc\" (UniqueName: \"kubernetes.io/projected/80c66ec7-5430-41a1-8641-56db5e4e9d73-kube-api-access-x4zhc\") pod \"80c66ec7-5430-41a1-8641-56db5e4e9d73\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " Nov 01 19:59:42.047158 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.046974 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80c66ec7-5430-41a1-8641-56db5e4e9d73-tls-assets\" (UniqueName: \"kubernetes.io/projected/80c66ec7-5430-41a1-8641-56db5e4e9d73-tls-assets\") pod \"80c66ec7-5430-41a1-8641-56db5e4e9d73\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " Nov 01 19:59:42.047158 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.047008 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80c66ec7-5430-41a1-8641-56db5e4e9d73-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/80c66ec7-5430-41a1-8641-56db5e4e9d73-prometheus-k8s-rulefiles-1\") pod \"80c66ec7-5430-41a1-8641-56db5e4e9d73\" (UID: \"80c66ec7-5430-41a1-8641-56db5e4e9d73\") " Nov 01 19:59:42.047600 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.047554 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80c66ec7-5430-41a1-8641-56db5e4e9d73-prometheus-k8s-rulefiles-1" pod "80c66ec7-5430-41a1-8641-56db5e4e9d73" (UID: "80c66ec7-5430-41a1-8641-56db5e4e9d73"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 19:59:42.047788 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.047753 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80c66ec7-5430-41a1-8641-56db5e4e9d73-prometheus-k8s-db" pod "80c66ec7-5430-41a1-8641-56db5e4e9d73" (UID: "80c66ec7-5430-41a1-8641-56db5e4e9d73"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 19:59:42.047944 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.047919 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80c66ec7-5430-41a1-8641-56db5e4e9d73-configmap-kubelet-serving-ca-bundle" pod "80c66ec7-5430-41a1-8641-56db5e4e9d73" (UID: "80c66ec7-5430-41a1-8641-56db5e4e9d73"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 19:59:42.048349 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.048325 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80c66ec7-5430-41a1-8641-56db5e4e9d73-configmap-metrics-client-ca" pod "80c66ec7-5430-41a1-8641-56db5e4e9d73" (UID: "80c66ec7-5430-41a1-8641-56db5e4e9d73"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 19:59:42.049107 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.049080 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80c66ec7-5430-41a1-8641-56db5e4e9d73-prometheus-trusted-ca-bundle" pod "80c66ec7-5430-41a1-8641-56db5e4e9d73" (UID: "80c66ec7-5430-41a1-8641-56db5e4e9d73"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 19:59:42.049468 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.049380 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80c66ec7-5430-41a1-8641-56db5e4e9d73-prometheus-k8s-rulefiles-0" pod "80c66ec7-5430-41a1-8641-56db5e4e9d73" (UID: "80c66ec7-5430-41a1-8641-56db5e4e9d73"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 19:59:42.049587 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.049493 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80c66ec7-5430-41a1-8641-56db5e4e9d73-prometheus-k8s-rulefiles-2" pod "80c66ec7-5430-41a1-8641-56db5e4e9d73" (UID: "80c66ec7-5430-41a1-8641-56db5e4e9d73"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 19:59:42.050554 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.050449 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-secret-prometheus-k8s-tls" pod "80c66ec7-5430-41a1-8641-56db5e4e9d73" (UID: "80c66ec7-5430-41a1-8641-56db5e4e9d73"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 19:59:42.050838 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.050749 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-secret-prometheus-k8s-kube-rbac-proxy-web" pod "80c66ec7-5430-41a1-8641-56db5e4e9d73" (UID: "80c66ec7-5430-41a1-8641-56db5e4e9d73"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 19:59:42.051595 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.051515 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80c66ec7-5430-41a1-8641-56db5e4e9d73-config-out" pod "80c66ec7-5430-41a1-8641-56db5e4e9d73" (UID: "80c66ec7-5430-41a1-8641-56db5e4e9d73"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 19:59:42.051685 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.051622 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80c66ec7-5430-41a1-8641-56db5e4e9d73-configmap-serving-certs-ca-bundle" pod "80c66ec7-5430-41a1-8641-56db5e4e9d73" (UID: "80c66ec7-5430-41a1-8641-56db5e4e9d73"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 19:59:42.051828 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.051798 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-secret-prometheus-k8s-thanos-sidecar-tls" pod "80c66ec7-5430-41a1-8641-56db5e4e9d73" (UID: "80c66ec7-5430-41a1-8641-56db5e4e9d73"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 19:59:42.051934 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.051877 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-secret-grpc-tls" pod "80c66ec7-5430-41a1-8641-56db5e4e9d73" (UID: "80c66ec7-5430-41a1-8641-56db5e4e9d73"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 19:59:42.052047 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.052018 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-config" pod "80c66ec7-5430-41a1-8641-56db5e4e9d73" (UID: "80c66ec7-5430-41a1-8641-56db5e4e9d73"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 19:59:42.052192 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.052168 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-secret-metrics-client-certs" pod "80c66ec7-5430-41a1-8641-56db5e4e9d73" (UID: "80c66ec7-5430-41a1-8641-56db5e4e9d73"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 19:59:42.052258 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.052230 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-thanos-prometheus-http-client-file" pod "80c66ec7-5430-41a1-8641-56db5e4e9d73" (UID: "80c66ec7-5430-41a1-8641-56db5e4e9d73"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 19:59:42.052465 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.052443 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-secret-kube-rbac-proxy" pod "80c66ec7-5430-41a1-8641-56db5e4e9d73" (UID: "80c66ec7-5430-41a1-8641-56db5e4e9d73"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 19:59:42.052814 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.052796 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80c66ec7-5430-41a1-8641-56db5e4e9d73-kube-api-access-x4zhc" pod "80c66ec7-5430-41a1-8641-56db5e4e9d73" (UID: "80c66ec7-5430-41a1-8641-56db5e4e9d73"). InnerVolumeSpecName "kube-api-access-x4zhc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 19:59:42.052989 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.052969 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80c66ec7-5430-41a1-8641-56db5e4e9d73-tls-assets" pod "80c66ec7-5430-41a1-8641-56db5e4e9d73" (UID: "80c66ec7-5430-41a1-8641-56db5e4e9d73"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 19:59:42.060532 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.060509 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-web-config" pod "80c66ec7-5430-41a1-8641-56db5e4e9d73" (UID: "80c66ec7-5430-41a1-8641-56db5e4e9d73"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 19:59:42.148052 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.148017 5935 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-secret-grpc-tls\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 19:59:42.148052 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.148048 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4zhc\" (UniqueName: \"kubernetes.io/projected/80c66ec7-5430-41a1-8641-56db5e4e9d73-kube-api-access-x4zhc\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 19:59:42.148052 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.148058 5935 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/80c66ec7-5430-41a1-8641-56db5e4e9d73-tls-assets\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 19:59:42.148303 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.148069 5935 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/80c66ec7-5430-41a1-8641-56db5e4e9d73-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 19:59:42.148303 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.148079 5935 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 19:59:42.148303 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.148089 5935 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80c66ec7-5430-41a1-8641-56db5e4e9d73-prometheus-trusted-ca-bundle\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 19:59:42.148303 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.148098 5935 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-secret-prometheus-k8s-tls\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 19:59:42.148303 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.148106 5935 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/80c66ec7-5430-41a1-8641-56db5e4e9d73-config-out\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 19:59:42.148303 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.148115 5935 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 19:59:42.148303 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.148125 5935 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-web-config\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 19:59:42.148303 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.148133 5935 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-config\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 19:59:42.148303 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.148142 5935 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/80c66ec7-5430-41a1-8641-56db5e4e9d73-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 19:59:42.148303 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.148150 5935 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-secret-metrics-client-certs\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 19:59:42.148303 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.148159 5935 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/80c66ec7-5430-41a1-8641-56db5e4e9d73-prometheus-k8s-db\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 19:59:42.148303 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.148168 5935 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80c66ec7-5430-41a1-8641-56db5e4e9d73-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 19:59:42.148303 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.148177 5935 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/80c66ec7-5430-41a1-8641-56db5e4e9d73-configmap-metrics-client-ca\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 19:59:42.148303 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.148185 5935 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-thanos-prometheus-http-client-file\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 19:59:42.148303 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.148194 5935 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/80c66ec7-5430-41a1-8641-56db5e4e9d73-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 19:59:42.148303 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.148202 5935 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80c66ec7-5430-41a1-8641-56db5e4e9d73-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 19:59:42.148303 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.148210 5935 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/80c66ec7-5430-41a1-8641-56db5e4e9d73-secret-kube-rbac-proxy\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 19:59:42.353392 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.351435 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87d7d5b7-5e96-4777-afcb-2a26d45b0ba8" path="/var/lib/kubelet/pods/87d7d5b7-5e96-4777-afcb-2a26d45b0ba8/volumes" Nov 01 19:59:42.520916 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.520888 5935 generic.go:356] "Generic (PLEG): container finished" podID="80c66ec7-5430-41a1-8641-56db5e4e9d73" containerID="c51258636e3bb432293783857dddfce6b61d33076f8dcaf32df54d4d9da2f10e" exitCode=0 Nov 01 19:59:42.521269 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.520930 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"80c66ec7-5430-41a1-8641-56db5e4e9d73","Type":"ContainerDied","Data":"c51258636e3bb432293783857dddfce6b61d33076f8dcaf32df54d4d9da2f10e"} Nov 01 19:59:42.521269 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.520983 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"80c66ec7-5430-41a1-8641-56db5e4e9d73","Type":"ContainerDied","Data":"21eeb2fddeba4c6a0109132af0db2fdc6635c8a149fcaa1750e43d852575ed2f"} Nov 01 19:59:42.521269 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.520995 5935 scope.go:122] "RemoveContainer" containerID="54513e5b3e839c8bdba0581043ed5eac858babb463bc57d6f83f597d437a96a2" Nov 01 19:59:42.521269 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.520981 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:42.530310 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.530276 5935 scope.go:122] "RemoveContainer" containerID="24713a72c5a0df1f0334a2d435e98bdb64cedb75a2385deb57868ffbb5c8f878" Nov 01 19:59:42.537136 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.537120 5935 scope.go:122] "RemoveContainer" containerID="517de8fdee23b80434e71946162632465d7c9580a7b69ade708ffdf401cdd144" Nov 01 19:59:42.543543 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.543524 5935 scope.go:122] "RemoveContainer" containerID="e25643c77686e3395636e0b4a10f7904a73836d08aeba79db39bfad411bb3f0d" Nov 01 19:59:42.544033 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.544015 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 19:59:42.548463 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.548440 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 19:59:42.550472 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.550454 5935 scope.go:122] "RemoveContainer" containerID="39ca1779d3e2325f0cc084bff04614b589b1f8e0c119fd669272dbd3c28dd02c" Nov 01 19:59:42.556586 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.556567 5935 scope.go:122] "RemoveContainer" containerID="c51258636e3bb432293783857dddfce6b61d33076f8dcaf32df54d4d9da2f10e" Nov 01 19:59:42.563075 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.563061 5935 scope.go:122] "RemoveContainer" containerID="86c3f1c47b655a4accd6f8c872ff24bcfd277f8e89dd328467251033cab7562f" Nov 01 19:59:42.569080 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.569064 5935 scope.go:122] "RemoveContainer" containerID="54513e5b3e839c8bdba0581043ed5eac858babb463bc57d6f83f597d437a96a2" Nov 01 19:59:42.569351 ip-10-0-136-26 kubenswrapper[5935]: E1101 19:59:42.569333 5935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54513e5b3e839c8bdba0581043ed5eac858babb463bc57d6f83f597d437a96a2\": container with ID starting with 54513e5b3e839c8bdba0581043ed5eac858babb463bc57d6f83f597d437a96a2 not found: ID does not exist" containerID="54513e5b3e839c8bdba0581043ed5eac858babb463bc57d6f83f597d437a96a2" Nov 01 19:59:42.569420 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.569358 5935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54513e5b3e839c8bdba0581043ed5eac858babb463bc57d6f83f597d437a96a2"} err="failed to get container status \"54513e5b3e839c8bdba0581043ed5eac858babb463bc57d6f83f597d437a96a2\": rpc error: code = NotFound desc = could not find container \"54513e5b3e839c8bdba0581043ed5eac858babb463bc57d6f83f597d437a96a2\": container with ID starting with 54513e5b3e839c8bdba0581043ed5eac858babb463bc57d6f83f597d437a96a2 not found: ID does not exist" Nov 01 19:59:42.569420 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.569387 5935 scope.go:122] "RemoveContainer" containerID="24713a72c5a0df1f0334a2d435e98bdb64cedb75a2385deb57868ffbb5c8f878" Nov 01 19:59:42.569599 ip-10-0-136-26 kubenswrapper[5935]: E1101 19:59:42.569584 5935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24713a72c5a0df1f0334a2d435e98bdb64cedb75a2385deb57868ffbb5c8f878\": container with ID starting with 24713a72c5a0df1f0334a2d435e98bdb64cedb75a2385deb57868ffbb5c8f878 not found: ID does not exist" containerID="24713a72c5a0df1f0334a2d435e98bdb64cedb75a2385deb57868ffbb5c8f878" Nov 01 19:59:42.569635 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.569603 5935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24713a72c5a0df1f0334a2d435e98bdb64cedb75a2385deb57868ffbb5c8f878"} err="failed to get container status \"24713a72c5a0df1f0334a2d435e98bdb64cedb75a2385deb57868ffbb5c8f878\": rpc error: code = NotFound desc = could not find container \"24713a72c5a0df1f0334a2d435e98bdb64cedb75a2385deb57868ffbb5c8f878\": container with ID starting with 24713a72c5a0df1f0334a2d435e98bdb64cedb75a2385deb57868ffbb5c8f878 not found: ID does not exist" Nov 01 19:59:42.569635 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.569615 5935 scope.go:122] "RemoveContainer" containerID="517de8fdee23b80434e71946162632465d7c9580a7b69ade708ffdf401cdd144" Nov 01 19:59:42.569801 ip-10-0-136-26 kubenswrapper[5935]: E1101 19:59:42.569786 5935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"517de8fdee23b80434e71946162632465d7c9580a7b69ade708ffdf401cdd144\": container with ID starting with 517de8fdee23b80434e71946162632465d7c9580a7b69ade708ffdf401cdd144 not found: ID does not exist" containerID="517de8fdee23b80434e71946162632465d7c9580a7b69ade708ffdf401cdd144" Nov 01 19:59:42.569842 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.569805 5935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"517de8fdee23b80434e71946162632465d7c9580a7b69ade708ffdf401cdd144"} err="failed to get container status \"517de8fdee23b80434e71946162632465d7c9580a7b69ade708ffdf401cdd144\": rpc error: code = NotFound desc = could not find container \"517de8fdee23b80434e71946162632465d7c9580a7b69ade708ffdf401cdd144\": container with ID starting with 517de8fdee23b80434e71946162632465d7c9580a7b69ade708ffdf401cdd144 not found: ID does not exist" Nov 01 19:59:42.569842 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.569818 5935 scope.go:122] "RemoveContainer" containerID="e25643c77686e3395636e0b4a10f7904a73836d08aeba79db39bfad411bb3f0d" Nov 01 19:59:42.570051 ip-10-0-136-26 kubenswrapper[5935]: E1101 19:59:42.570036 5935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e25643c77686e3395636e0b4a10f7904a73836d08aeba79db39bfad411bb3f0d\": container with ID starting with e25643c77686e3395636e0b4a10f7904a73836d08aeba79db39bfad411bb3f0d not found: ID does not exist" containerID="e25643c77686e3395636e0b4a10f7904a73836d08aeba79db39bfad411bb3f0d" Nov 01 19:59:42.570090 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.570054 5935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e25643c77686e3395636e0b4a10f7904a73836d08aeba79db39bfad411bb3f0d"} err="failed to get container status \"e25643c77686e3395636e0b4a10f7904a73836d08aeba79db39bfad411bb3f0d\": rpc error: code = NotFound desc = could not find container \"e25643c77686e3395636e0b4a10f7904a73836d08aeba79db39bfad411bb3f0d\": container with ID starting with e25643c77686e3395636e0b4a10f7904a73836d08aeba79db39bfad411bb3f0d not found: ID does not exist" Nov 01 19:59:42.570090 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.570067 5935 scope.go:122] "RemoveContainer" containerID="39ca1779d3e2325f0cc084bff04614b589b1f8e0c119fd669272dbd3c28dd02c" Nov 01 19:59:42.570250 ip-10-0-136-26 kubenswrapper[5935]: E1101 19:59:42.570237 5935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39ca1779d3e2325f0cc084bff04614b589b1f8e0c119fd669272dbd3c28dd02c\": container with ID starting with 39ca1779d3e2325f0cc084bff04614b589b1f8e0c119fd669272dbd3c28dd02c not found: ID does not exist" containerID="39ca1779d3e2325f0cc084bff04614b589b1f8e0c119fd669272dbd3c28dd02c" Nov 01 19:59:42.570299 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.570253 5935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39ca1779d3e2325f0cc084bff04614b589b1f8e0c119fd669272dbd3c28dd02c"} err="failed to get container status \"39ca1779d3e2325f0cc084bff04614b589b1f8e0c119fd669272dbd3c28dd02c\": rpc error: code = NotFound desc = could not find container \"39ca1779d3e2325f0cc084bff04614b589b1f8e0c119fd669272dbd3c28dd02c\": container with ID starting with 39ca1779d3e2325f0cc084bff04614b589b1f8e0c119fd669272dbd3c28dd02c not found: ID does not exist" Nov 01 19:59:42.570299 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.570265 5935 scope.go:122] "RemoveContainer" containerID="c51258636e3bb432293783857dddfce6b61d33076f8dcaf32df54d4d9da2f10e" Nov 01 19:59:42.570486 ip-10-0-136-26 kubenswrapper[5935]: E1101 19:59:42.570466 5935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c51258636e3bb432293783857dddfce6b61d33076f8dcaf32df54d4d9da2f10e\": container with ID starting with c51258636e3bb432293783857dddfce6b61d33076f8dcaf32df54d4d9da2f10e not found: ID does not exist" containerID="c51258636e3bb432293783857dddfce6b61d33076f8dcaf32df54d4d9da2f10e" Nov 01 19:59:42.570524 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.570489 5935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c51258636e3bb432293783857dddfce6b61d33076f8dcaf32df54d4d9da2f10e"} err="failed to get container status \"c51258636e3bb432293783857dddfce6b61d33076f8dcaf32df54d4d9da2f10e\": rpc error: code = NotFound desc = could not find container \"c51258636e3bb432293783857dddfce6b61d33076f8dcaf32df54d4d9da2f10e\": container with ID starting with c51258636e3bb432293783857dddfce6b61d33076f8dcaf32df54d4d9da2f10e not found: ID does not exist" Nov 01 19:59:42.570524 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.570500 5935 scope.go:122] "RemoveContainer" containerID="86c3f1c47b655a4accd6f8c872ff24bcfd277f8e89dd328467251033cab7562f" Nov 01 19:59:42.570696 ip-10-0-136-26 kubenswrapper[5935]: E1101 19:59:42.570682 5935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86c3f1c47b655a4accd6f8c872ff24bcfd277f8e89dd328467251033cab7562f\": container with ID starting with 86c3f1c47b655a4accd6f8c872ff24bcfd277f8e89dd328467251033cab7562f not found: ID does not exist" containerID="86c3f1c47b655a4accd6f8c872ff24bcfd277f8e89dd328467251033cab7562f" Nov 01 19:59:42.570733 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.570698 5935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86c3f1c47b655a4accd6f8c872ff24bcfd277f8e89dd328467251033cab7562f"} err="failed to get container status \"86c3f1c47b655a4accd6f8c872ff24bcfd277f8e89dd328467251033cab7562f\": rpc error: code = NotFound desc = could not find container \"86c3f1c47b655a4accd6f8c872ff24bcfd277f8e89dd328467251033cab7562f\": container with ID starting with 86c3f1c47b655a4accd6f8c872ff24bcfd277f8e89dd328467251033cab7562f not found: ID does not exist" Nov 01 19:59:42.575022 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.575001 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 19:59:42.575281 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.575267 5935 cpu_manager.go:414] "containerMap: removing container" podUID="80c66ec7-5430-41a1-8641-56db5e4e9d73" containerName="init-config-reloader" Nov 01 19:59:42.575346 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.575297 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80c66ec7-5430-41a1-8641-56db5e4e9d73" containerName="init-config-reloader" Nov 01 19:59:42.575346 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.575317 5935 cpu_manager.go:414] "containerMap: removing container" podUID="80c66ec7-5430-41a1-8641-56db5e4e9d73" containerName="thanos-sidecar" Nov 01 19:59:42.575346 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.575322 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80c66ec7-5430-41a1-8641-56db5e4e9d73" containerName="thanos-sidecar" Nov 01 19:59:42.575346 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.575329 5935 cpu_manager.go:414] "containerMap: removing container" podUID="87d7d5b7-5e96-4777-afcb-2a26d45b0ba8" containerName="console" Nov 01 19:59:42.575346 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.575334 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87d7d5b7-5e96-4777-afcb-2a26d45b0ba8" containerName="console" Nov 01 19:59:42.575529 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.575347 5935 cpu_manager.go:414] "containerMap: removing container" podUID="80c66ec7-5430-41a1-8641-56db5e4e9d73" containerName="prometheus" Nov 01 19:59:42.575529 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.575354 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80c66ec7-5430-41a1-8641-56db5e4e9d73" containerName="prometheus" Nov 01 19:59:42.575529 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.575366 5935 cpu_manager.go:414] "containerMap: removing container" podUID="80c66ec7-5430-41a1-8641-56db5e4e9d73" containerName="config-reloader" Nov 01 19:59:42.575529 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.575372 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80c66ec7-5430-41a1-8641-56db5e4e9d73" containerName="config-reloader" Nov 01 19:59:42.575529 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.575381 5935 cpu_manager.go:414] "containerMap: removing container" podUID="80c66ec7-5430-41a1-8641-56db5e4e9d73" containerName="kube-rbac-proxy" Nov 01 19:59:42.575529 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.575386 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80c66ec7-5430-41a1-8641-56db5e4e9d73" containerName="kube-rbac-proxy" Nov 01 19:59:42.575529 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.575394 5935 cpu_manager.go:414] "containerMap: removing container" podUID="80c66ec7-5430-41a1-8641-56db5e4e9d73" containerName="kube-rbac-proxy-web" Nov 01 19:59:42.575529 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.575398 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80c66ec7-5430-41a1-8641-56db5e4e9d73" containerName="kube-rbac-proxy-web" Nov 01 19:59:42.575529 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.575408 5935 cpu_manager.go:414] "containerMap: removing container" podUID="80c66ec7-5430-41a1-8641-56db5e4e9d73" containerName="kube-rbac-proxy-thanos" Nov 01 19:59:42.575529 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.575412 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80c66ec7-5430-41a1-8641-56db5e4e9d73" containerName="kube-rbac-proxy-thanos" Nov 01 19:59:42.575529 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.575455 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="80c66ec7-5430-41a1-8641-56db5e4e9d73" containerName="thanos-sidecar" Nov 01 19:59:42.575529 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.575463 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="87d7d5b7-5e96-4777-afcb-2a26d45b0ba8" containerName="console" Nov 01 19:59:42.575529 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.575469 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="80c66ec7-5430-41a1-8641-56db5e4e9d73" containerName="prometheus" Nov 01 19:59:42.575529 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.575474 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="80c66ec7-5430-41a1-8641-56db5e4e9d73" containerName="config-reloader" Nov 01 19:59:42.575529 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.575479 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="80c66ec7-5430-41a1-8641-56db5e4e9d73" containerName="kube-rbac-proxy" Nov 01 19:59:42.575529 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.575485 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="80c66ec7-5430-41a1-8641-56db5e4e9d73" containerName="kube-rbac-proxy-web" Nov 01 19:59:42.575529 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.575490 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="80c66ec7-5430-41a1-8641-56db5e4e9d73" containerName="kube-rbac-proxy-thanos" Nov 01 19:59:42.575529 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.575495 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="80c66ec7-5430-41a1-8641-56db5e4e9d73" containerName="init-config-reloader" Nov 01 19:59:42.578686 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.578670 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:42.582532 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.582507 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Nov 01 19:59:42.582617 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.582547 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Nov 01 19:59:42.582617 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.582566 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Nov 01 19:59:42.582979 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.582956 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Nov 01 19:59:42.583428 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.583410 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Nov 01 19:59:42.583525 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.583513 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Nov 01 19:59:42.584058 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.584039 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Nov 01 19:59:42.584058 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.584047 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Nov 01 19:59:42.584212 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.584074 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Nov 01 19:59:42.584875 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.584813 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Nov 01 19:59:42.584875 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.584841 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-dsvm4iuppcnu7\"" Nov 01 19:59:42.584875 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.584857 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Nov 01 19:59:42.585017 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.584888 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Nov 01 19:59:42.585017 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.584935 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Nov 01 19:59:42.587933 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.587918 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-ns62c\"" Nov 01 19:59:42.589342 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.589325 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Nov 01 19:59:42.589447 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.589432 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Nov 01 19:59:42.594379 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.594358 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 19:59:42.753536 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.753509 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:42.753646 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.753540 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:42.753646 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.753560 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:42.753646 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.753633 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:42.753760 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.753689 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:42.753802 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.753765 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:42.753802 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.753784 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:42.753864 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.753804 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:42.753864 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.753857 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9-config-out\") pod \"prometheus-k8s-0\" (UID: \"4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:42.753937 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.753886 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:42.753979 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.753935 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:42.753979 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.753960 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9-config\") pod \"prometheus-k8s-0\" (UID: \"4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:42.753979 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.753976 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:42.754083 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.754007 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjmql\" (UniqueName: \"kubernetes.io/projected/4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9-kube-api-access-hjmql\") pod \"prometheus-k8s-0\" (UID: \"4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:42.754083 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.754052 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:42.754083 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.754078 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:42.754181 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.754096 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9-web-config\") pod \"prometheus-k8s-0\" (UID: \"4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:42.754181 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.754116 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:42.754181 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.754135 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:42.754181 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.754158 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:42.855066 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.855042 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:42.855155 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.855068 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:42.855155 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.855086 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:42.855155 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.855113 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:42.855155 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.855138 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:42.855385 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.855162 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:42.855385 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.855190 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:42.855385 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.855216 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:42.855385 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.855246 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:42.855385 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.855271 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:42.855385 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.855324 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:42.855385 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.855364 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9-config-out\") pod \"prometheus-k8s-0\" (UID: \"4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:42.855727 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.855395 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:42.855727 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.855425 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:42.855727 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.855453 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9-config\") pod \"prometheus-k8s-0\" (UID: \"4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:42.855727 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.855477 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:42.855727 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.855504 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjmql\" (UniqueName: \"kubernetes.io/projected/4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9-kube-api-access-hjmql\") pod \"prometheus-k8s-0\" (UID: \"4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:42.855727 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.855537 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:42.855727 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.855569 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:42.855727 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.855595 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9-web-config\") pod \"prometheus-k8s-0\" (UID: \"4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:42.856103 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.855931 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:42.856326 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.856209 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:42.857098 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.856343 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:42.857098 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.856785 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:42.857098 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.856813 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:42.857098 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.857091 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:42.857792 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.857763 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:42.858130 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.858053 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:42.859387 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.859333 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:42.859922 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.859894 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:42.860023 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.859971 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:42.860592 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.860549 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:42.860696 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.860639 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9-config\") pod \"prometheus-k8s-0\" (UID: \"4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:42.860779 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.860754 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9-web-config\") pod \"prometheus-k8s-0\" (UID: \"4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:42.860843 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.860797 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:42.861272 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.861247 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:42.861482 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.861462 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:42.862086 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.862063 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9-config-out\") pod \"prometheus-k8s-0\" (UID: \"4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:42.862160 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.862141 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:42.864191 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.864171 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjmql\" (UniqueName: \"kubernetes.io/projected/4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9-kube-api-access-hjmql\") pod \"prometheus-k8s-0\" (UID: \"4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9\") " pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:42.896191 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:42.896173 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 19:59:43.175277 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:43.175249 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Nov 01 19:59:43.176825 ip-10-0-136-26 kubenswrapper[5935]: W1101 19:59:43.176802 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d7a2e00_cce9_4bc2_baf8_a27bfe9107b9.slice/crio-0c9cddcf51e59405ccafda5866cd83fc4303b340bd1799d8d005da89fe8394b3 WatchSource:0}: Error finding container 0c9cddcf51e59405ccafda5866cd83fc4303b340bd1799d8d005da89fe8394b3: Status 404 returned error can't find the container with id 0c9cddcf51e59405ccafda5866cd83fc4303b340bd1799d8d005da89fe8394b3 Nov 01 19:59:43.526100 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:43.526067 5935 generic.go:356] "Generic (PLEG): container finished" podID="4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9" containerID="fa52fb9609fa370954008c10729d8b1d0029f404eaa563c3f020c5035303df81" exitCode=0 Nov 01 19:59:43.526470 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:43.526138 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9","Type":"ContainerDied","Data":"fa52fb9609fa370954008c10729d8b1d0029f404eaa563c3f020c5035303df81"} Nov 01 19:59:43.526470 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:43.526203 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9","Type":"ContainerStarted","Data":"0c9cddcf51e59405ccafda5866cd83fc4303b340bd1799d8d005da89fe8394b3"} Nov 01 19:59:43.574936 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:43.574916 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5fccf54c48-xhbhp" Nov 01 19:59:43.575026 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:43.574964 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5fccf54c48-xhbhp" Nov 01 19:59:43.580156 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:43.580134 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5fccf54c48-xhbhp" Nov 01 19:59:43.591398 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:43.591376 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-b7mcz" Nov 01 19:59:44.350127 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:44.350094 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80c66ec7-5430-41a1-8641-56db5e4e9d73" path="/var/lib/kubelet/pods/80c66ec7-5430-41a1-8641-56db5e4e9d73/volumes" Nov 01 19:59:44.533001 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:44.532964 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9","Type":"ContainerStarted","Data":"4c4c2ba8bc064415b45c9a517f918226747e6e8a30d745ae5640b1f5bc3f7a61"} Nov 01 19:59:44.533499 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:44.533025 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9","Type":"ContainerStarted","Data":"56f06d874798d69a34b40a40ef009ab29069eb9580e88e962b2f3fe0ebb88abe"} Nov 01 19:59:44.533499 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:44.533058 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9","Type":"ContainerStarted","Data":"76f0fcdd360c413b9921c3dda853fcbcd1ba4b392da1e3ad196ca011862356f1"} Nov 01 19:59:44.533499 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:44.533082 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9","Type":"ContainerStarted","Data":"2c24597b754ddac92707a8b66b678097e00ff10e9d4922e0a5836d6d0e897008"} Nov 01 19:59:44.533499 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:44.533110 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9","Type":"ContainerStarted","Data":"4ef7912fb6b8da7e652a6baed63ae1915cf6c14c3ae782aae8f98440bb1dc008"} Nov 01 19:59:44.533499 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:44.533133 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9","Type":"ContainerStarted","Data":"cb566b8d07b9fc480d27b04809e608650f796c45f6fba00940c5ff21a5ac7312"} Nov 01 19:59:44.536832 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:44.536810 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5fccf54c48-xhbhp" Nov 01 19:59:44.560689 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:44.560590 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.545686726 podStartE2EDuration="2.560563386s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.87666ms" isStatefulPod=false podCreationTimestamp="2026-11-01 19:59:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 19:59:44.558815138 +0000 UTC m=+30.331975650" watchObservedRunningTime="2026-11-01 19:59:44.560563386 +0000 UTC m=+30.333723929" Nov 01 19:59:44.601276 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:44.601214 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7698dd44-8ddgn"] Nov 01 19:59:47.896786 ip-10-0-136-26 kubenswrapper[5935]: I1101 19:59:47.896754 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:00:05.842061 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:00:05.842023 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:00:09.619962 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:00:09.619922 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7698dd44-8ddgn" podUID="60e3a171-27e7-4ae6-ad6c-89984ecf5634" containerName="console" containerID="cri-o://deab280cd317ce9dd53293758118ab4fe5e903b9ba385bc1baae6fac7a5a1a37" gracePeriod=15 Nov 01 20:00:09.853444 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:00:09.853422 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7698dd44-8ddgn" Nov 01 20:00:09.961854 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:00:09.961833 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60e3a171-27e7-4ae6-ad6c-89984ecf5634-kube-api-access-4cq6h\" (UniqueName: \"kubernetes.io/projected/60e3a171-27e7-4ae6-ad6c-89984ecf5634-kube-api-access-4cq6h\") pod \"60e3a171-27e7-4ae6-ad6c-89984ecf5634\" (UID: \"60e3a171-27e7-4ae6-ad6c-89984ecf5634\") " Nov 01 20:00:09.962003 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:00:09.961863 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60e3a171-27e7-4ae6-ad6c-89984ecf5634-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/60e3a171-27e7-4ae6-ad6c-89984ecf5634-oauth-serving-cert\") pod \"60e3a171-27e7-4ae6-ad6c-89984ecf5634\" (UID: \"60e3a171-27e7-4ae6-ad6c-89984ecf5634\") " Nov 01 20:00:09.962003 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:00:09.961897 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60e3a171-27e7-4ae6-ad6c-89984ecf5634-tmp\" (UniqueName: \"kubernetes.io/empty-dir/60e3a171-27e7-4ae6-ad6c-89984ecf5634-tmp\") pod \"60e3a171-27e7-4ae6-ad6c-89984ecf5634\" (UID: \"60e3a171-27e7-4ae6-ad6c-89984ecf5634\") " Nov 01 20:00:09.962003 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:00:09.961920 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60e3a171-27e7-4ae6-ad6c-89984ecf5634-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/60e3a171-27e7-4ae6-ad6c-89984ecf5634-console-serving-cert\") pod \"60e3a171-27e7-4ae6-ad6c-89984ecf5634\" (UID: \"60e3a171-27e7-4ae6-ad6c-89984ecf5634\") " Nov 01 20:00:09.962003 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:00:09.961958 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60e3a171-27e7-4ae6-ad6c-89984ecf5634-console-config\" (UniqueName: \"kubernetes.io/configmap/60e3a171-27e7-4ae6-ad6c-89984ecf5634-console-config\") pod \"60e3a171-27e7-4ae6-ad6c-89984ecf5634\" (UID: \"60e3a171-27e7-4ae6-ad6c-89984ecf5634\") " Nov 01 20:00:09.962003 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:00:09.961988 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60e3a171-27e7-4ae6-ad6c-89984ecf5634-service-ca\" (UniqueName: \"kubernetes.io/configmap/60e3a171-27e7-4ae6-ad6c-89984ecf5634-service-ca\") pod \"60e3a171-27e7-4ae6-ad6c-89984ecf5634\" (UID: \"60e3a171-27e7-4ae6-ad6c-89984ecf5634\") " Nov 01 20:00:09.962230 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:00:09.962123 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60e3a171-27e7-4ae6-ad6c-89984ecf5634-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/60e3a171-27e7-4ae6-ad6c-89984ecf5634-console-oauth-config\") pod \"60e3a171-27e7-4ae6-ad6c-89984ecf5634\" (UID: \"60e3a171-27e7-4ae6-ad6c-89984ecf5634\") " Nov 01 20:00:09.962230 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:00:09.962168 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60e3a171-27e7-4ae6-ad6c-89984ecf5634-tmp" pod "60e3a171-27e7-4ae6-ad6c-89984ecf5634" (UID: "60e3a171-27e7-4ae6-ad6c-89984ecf5634"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:00:09.962230 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:00:09.962190 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60e3a171-27e7-4ae6-ad6c-89984ecf5634-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60e3a171-27e7-4ae6-ad6c-89984ecf5634-trusted-ca-bundle\") pod \"60e3a171-27e7-4ae6-ad6c-89984ecf5634\" (UID: \"60e3a171-27e7-4ae6-ad6c-89984ecf5634\") " Nov 01 20:00:09.962414 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:00:09.962375 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60e3a171-27e7-4ae6-ad6c-89984ecf5634-oauth-serving-cert" pod "60e3a171-27e7-4ae6-ad6c-89984ecf5634" (UID: "60e3a171-27e7-4ae6-ad6c-89984ecf5634"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:00:09.962470 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:00:09.962424 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60e3a171-27e7-4ae6-ad6c-89984ecf5634-service-ca" pod "60e3a171-27e7-4ae6-ad6c-89984ecf5634" (UID: "60e3a171-27e7-4ae6-ad6c-89984ecf5634"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:00:09.962470 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:00:09.962446 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60e3a171-27e7-4ae6-ad6c-89984ecf5634-console-config" pod "60e3a171-27e7-4ae6-ad6c-89984ecf5634" (UID: "60e3a171-27e7-4ae6-ad6c-89984ecf5634"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:00:09.962470 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:00:09.962461 5935 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/60e3a171-27e7-4ae6-ad6c-89984ecf5634-oauth-serving-cert\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:00:09.962599 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:00:09.962480 5935 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/60e3a171-27e7-4ae6-ad6c-89984ecf5634-tmp\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:00:09.962745 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:00:09.962717 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60e3a171-27e7-4ae6-ad6c-89984ecf5634-trusted-ca-bundle" pod "60e3a171-27e7-4ae6-ad6c-89984ecf5634" (UID: "60e3a171-27e7-4ae6-ad6c-89984ecf5634"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:00:09.964214 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:00:09.964186 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60e3a171-27e7-4ae6-ad6c-89984ecf5634-console-serving-cert" pod "60e3a171-27e7-4ae6-ad6c-89984ecf5634" (UID: "60e3a171-27e7-4ae6-ad6c-89984ecf5634"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:00:09.964312 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:00:09.964226 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60e3a171-27e7-4ae6-ad6c-89984ecf5634-kube-api-access-4cq6h" pod "60e3a171-27e7-4ae6-ad6c-89984ecf5634" (UID: "60e3a171-27e7-4ae6-ad6c-89984ecf5634"). InnerVolumeSpecName "kube-api-access-4cq6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:00:09.964359 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:00:09.964345 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60e3a171-27e7-4ae6-ad6c-89984ecf5634-console-oauth-config" pod "60e3a171-27e7-4ae6-ad6c-89984ecf5634" (UID: "60e3a171-27e7-4ae6-ad6c-89984ecf5634"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:00:10.063766 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:00:10.063739 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cq6h\" (UniqueName: \"kubernetes.io/projected/60e3a171-27e7-4ae6-ad6c-89984ecf5634-kube-api-access-4cq6h\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:00:10.063766 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:00:10.063762 5935 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/60e3a171-27e7-4ae6-ad6c-89984ecf5634-console-serving-cert\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:00:10.063903 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:00:10.063773 5935 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/60e3a171-27e7-4ae6-ad6c-89984ecf5634-console-config\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:00:10.063903 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:00:10.063782 5935 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/60e3a171-27e7-4ae6-ad6c-89984ecf5634-service-ca\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:00:10.063903 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:00:10.063791 5935 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/60e3a171-27e7-4ae6-ad6c-89984ecf5634-console-oauth-config\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:00:10.063903 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:00:10.063800 5935 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60e3a171-27e7-4ae6-ad6c-89984ecf5634-trusted-ca-bundle\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:00:10.621904 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:00:10.621822 5935 generic.go:356] "Generic (PLEG): container finished" podID="60e3a171-27e7-4ae6-ad6c-89984ecf5634" containerID="deab280cd317ce9dd53293758118ab4fe5e903b9ba385bc1baae6fac7a5a1a37" exitCode=0 Nov 01 20:00:10.621904 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:00:10.621870 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7698dd44-8ddgn" Nov 01 20:00:10.622434 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:00:10.621903 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7698dd44-8ddgn" event={"ID":"60e3a171-27e7-4ae6-ad6c-89984ecf5634","Type":"ContainerDied","Data":"deab280cd317ce9dd53293758118ab4fe5e903b9ba385bc1baae6fac7a5a1a37"} Nov 01 20:00:10.622434 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:00:10.621964 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7698dd44-8ddgn" event={"ID":"60e3a171-27e7-4ae6-ad6c-89984ecf5634","Type":"ContainerDied","Data":"b3ea44655cb46e9c5ef929ad76a73cb9490badcb9c5405bde8f151cfa4aeb8c8"} Nov 01 20:00:10.622434 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:00:10.621985 5935 scope.go:122] "RemoveContainer" containerID="deab280cd317ce9dd53293758118ab4fe5e903b9ba385bc1baae6fac7a5a1a37" Nov 01 20:00:10.629997 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:00:10.629980 5935 scope.go:122] "RemoveContainer" containerID="deab280cd317ce9dd53293758118ab4fe5e903b9ba385bc1baae6fac7a5a1a37" Nov 01 20:00:10.630265 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:00:10.630247 5935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"deab280cd317ce9dd53293758118ab4fe5e903b9ba385bc1baae6fac7a5a1a37\": container with ID starting with deab280cd317ce9dd53293758118ab4fe5e903b9ba385bc1baae6fac7a5a1a37 not found: ID does not exist" containerID="deab280cd317ce9dd53293758118ab4fe5e903b9ba385bc1baae6fac7a5a1a37" Nov 01 20:00:10.630345 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:00:10.630278 5935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"deab280cd317ce9dd53293758118ab4fe5e903b9ba385bc1baae6fac7a5a1a37"} err="failed to get container status \"deab280cd317ce9dd53293758118ab4fe5e903b9ba385bc1baae6fac7a5a1a37\": rpc error: code = NotFound desc = could not find container \"deab280cd317ce9dd53293758118ab4fe5e903b9ba385bc1baae6fac7a5a1a37\": container with ID starting with deab280cd317ce9dd53293758118ab4fe5e903b9ba385bc1baae6fac7a5a1a37 not found: ID does not exist" Nov 01 20:00:10.638299 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:00:10.638261 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7698dd44-8ddgn"] Nov 01 20:00:10.641924 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:00:10.641903 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7698dd44-8ddgn"] Nov 01 20:00:12.350270 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:00:12.350239 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60e3a171-27e7-4ae6-ad6c-89984ecf5634" path="/var/lib/kubelet/pods/60e3a171-27e7-4ae6-ad6c-89984ecf5634/volumes" Nov 01 20:00:17.316643 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:00:17.316612 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:00:17.331606 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:00:17.331578 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:00:17.662440 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:00:17.662363 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Nov 01 20:00:35.841074 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:00:35.841035 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:01:02.724684 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:02.724649 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atdgc9"] Nov 01 20:01:02.725268 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:02.725032 5935 cpu_manager.go:414] "containerMap: removing container" podUID="60e3a171-27e7-4ae6-ad6c-89984ecf5634" containerName="console" Nov 01 20:01:02.725268 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:02.725049 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60e3a171-27e7-4ae6-ad6c-89984ecf5634" containerName="console" Nov 01 20:01:02.725268 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:02.725182 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="60e3a171-27e7-4ae6-ad6c-89984ecf5634" containerName="console" Nov 01 20:01:02.728127 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:02.728105 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atdgc9" Nov 01 20:01:02.736109 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:02.736088 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-vzccx\"" Nov 01 20:01:02.736333 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:02.736181 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 01 20:01:02.737052 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:02.737029 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atdgc9"] Nov 01 20:01:02.737254 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:02.737222 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 01 20:01:02.761056 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:02.761032 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2700c53f-9630-4fba-b2c3-ee481ecd5062-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atdgc9\" (UID: \"2700c53f-9630-4fba-b2c3-ee481ecd5062\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atdgc9" Nov 01 20:01:02.761185 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:02.761071 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br8bm\" (UniqueName: \"kubernetes.io/projected/2700c53f-9630-4fba-b2c3-ee481ecd5062-kube-api-access-br8bm\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atdgc9\" (UID: \"2700c53f-9630-4fba-b2c3-ee481ecd5062\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atdgc9" Nov 01 20:01:02.761185 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:02.761095 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2700c53f-9630-4fba-b2c3-ee481ecd5062-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atdgc9\" (UID: \"2700c53f-9630-4fba-b2c3-ee481ecd5062\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atdgc9" Nov 01 20:01:02.830057 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:02.830030 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78v7xd"] Nov 01 20:01:02.833508 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:02.833493 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78v7xd" Nov 01 20:01:02.837472 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:02.837448 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-5djzp\"" Nov 01 20:01:02.841236 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:02.841215 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78v7xd"] Nov 01 20:01:02.861912 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:02.861891 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cf4f598f-731b-4c35-85b7-86f0e700e546-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78v7xd\" (UID: \"cf4f598f-731b-4c35-85b7-86f0e700e546\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78v7xd" Nov 01 20:01:02.862021 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:02.861931 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2700c53f-9630-4fba-b2c3-ee481ecd5062-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atdgc9\" (UID: \"2700c53f-9630-4fba-b2c3-ee481ecd5062\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atdgc9" Nov 01 20:01:02.862021 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:02.862004 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cf4f598f-731b-4c35-85b7-86f0e700e546-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78v7xd\" (UID: \"cf4f598f-731b-4c35-85b7-86f0e700e546\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78v7xd" Nov 01 20:01:02.862131 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:02.862034 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br8bm\" (UniqueName: \"kubernetes.io/projected/2700c53f-9630-4fba-b2c3-ee481ecd5062-kube-api-access-br8bm\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atdgc9\" (UID: \"2700c53f-9630-4fba-b2c3-ee481ecd5062\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atdgc9" Nov 01 20:01:02.862131 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:02.862094 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plmj2\" (UniqueName: \"kubernetes.io/projected/cf4f598f-731b-4c35-85b7-86f0e700e546-kube-api-access-plmj2\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78v7xd\" (UID: \"cf4f598f-731b-4c35-85b7-86f0e700e546\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78v7xd" Nov 01 20:01:02.862235 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:02.862138 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2700c53f-9630-4fba-b2c3-ee481ecd5062-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atdgc9\" (UID: \"2700c53f-9630-4fba-b2c3-ee481ecd5062\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atdgc9" Nov 01 20:01:02.862408 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:02.862388 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2700c53f-9630-4fba-b2c3-ee481ecd5062-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atdgc9\" (UID: \"2700c53f-9630-4fba-b2c3-ee481ecd5062\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atdgc9" Nov 01 20:01:02.862494 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:02.862474 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2700c53f-9630-4fba-b2c3-ee481ecd5062-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atdgc9\" (UID: \"2700c53f-9630-4fba-b2c3-ee481ecd5062\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atdgc9" Nov 01 20:01:02.869643 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:02.869620 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-br8bm\" (UniqueName: \"kubernetes.io/projected/2700c53f-9630-4fba-b2c3-ee481ecd5062-kube-api-access-br8bm\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atdgc9\" (UID: \"2700c53f-9630-4fba-b2c3-ee481ecd5062\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atdgc9" Nov 01 20:01:02.930331 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:02.930308 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7gdcm"] Nov 01 20:01:02.933764 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:02.933750 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7gdcm" Nov 01 20:01:02.936986 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:02.936968 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-xvn5p\"" Nov 01 20:01:02.941561 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:02.941542 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7gdcm"] Nov 01 20:01:02.963434 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:02.963412 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cww6r\" (UniqueName: \"kubernetes.io/projected/58945644-f546-451e-bd8f-a3236df037f9-kube-api-access-cww6r\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7gdcm\" (UID: \"58945644-f546-451e-bd8f-a3236df037f9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7gdcm" Nov 01 20:01:02.963531 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:02.963445 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cf4f598f-731b-4c35-85b7-86f0e700e546-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78v7xd\" (UID: \"cf4f598f-731b-4c35-85b7-86f0e700e546\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78v7xd" Nov 01 20:01:02.963531 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:02.963463 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/58945644-f546-451e-bd8f-a3236df037f9-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7gdcm\" (UID: \"58945644-f546-451e-bd8f-a3236df037f9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7gdcm" Nov 01 20:01:02.963602 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:02.963585 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cf4f598f-731b-4c35-85b7-86f0e700e546-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78v7xd\" (UID: \"cf4f598f-731b-4c35-85b7-86f0e700e546\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78v7xd" Nov 01 20:01:02.963638 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:02.963625 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plmj2\" (UniqueName: \"kubernetes.io/projected/cf4f598f-731b-4c35-85b7-86f0e700e546-kube-api-access-plmj2\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78v7xd\" (UID: \"cf4f598f-731b-4c35-85b7-86f0e700e546\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78v7xd" Nov 01 20:01:02.963696 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:02.963679 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/58945644-f546-451e-bd8f-a3236df037f9-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7gdcm\" (UID: \"58945644-f546-451e-bd8f-a3236df037f9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7gdcm" Nov 01 20:01:02.963836 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:02.963822 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cf4f598f-731b-4c35-85b7-86f0e700e546-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78v7xd\" (UID: \"cf4f598f-731b-4c35-85b7-86f0e700e546\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78v7xd" Nov 01 20:01:02.963876 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:02.963851 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cf4f598f-731b-4c35-85b7-86f0e700e546-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78v7xd\" (UID: \"cf4f598f-731b-4c35-85b7-86f0e700e546\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78v7xd" Nov 01 20:01:02.970919 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:02.970902 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plmj2\" (UniqueName: \"kubernetes.io/projected/cf4f598f-731b-4c35-85b7-86f0e700e546-kube-api-access-plmj2\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78v7xd\" (UID: \"cf4f598f-731b-4c35-85b7-86f0e700e546\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78v7xd" Nov 01 20:01:03.029598 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:03.029551 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lnfpz"] Nov 01 20:01:03.033059 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:03.033044 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lnfpz" Nov 01 20:01:03.036206 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:03.036191 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-4nxrp\"" Nov 01 20:01:03.040506 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:03.040484 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lnfpz"] Nov 01 20:01:03.045648 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:03.045632 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atdgc9" Nov 01 20:01:03.064416 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:03.064394 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/58945644-f546-451e-bd8f-a3236df037f9-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7gdcm\" (UID: \"58945644-f546-451e-bd8f-a3236df037f9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7gdcm" Nov 01 20:01:03.064516 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:03.064459 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvkpb\" (UniqueName: \"kubernetes.io/projected/aafb7b29-b974-46fa-9871-c9fbc188bb5f-kube-api-access-lvkpb\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lnfpz\" (UID: \"aafb7b29-b974-46fa-9871-c9fbc188bb5f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lnfpz" Nov 01 20:01:03.064516 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:03.064493 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aafb7b29-b974-46fa-9871-c9fbc188bb5f-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lnfpz\" (UID: \"aafb7b29-b974-46fa-9871-c9fbc188bb5f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lnfpz" Nov 01 20:01:03.064613 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:03.064526 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aafb7b29-b974-46fa-9871-c9fbc188bb5f-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lnfpz\" (UID: \"aafb7b29-b974-46fa-9871-c9fbc188bb5f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lnfpz" Nov 01 20:01:03.064613 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:03.064559 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/58945644-f546-451e-bd8f-a3236df037f9-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7gdcm\" (UID: \"58945644-f546-451e-bd8f-a3236df037f9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7gdcm" Nov 01 20:01:03.064727 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:03.064705 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cww6r\" (UniqueName: \"kubernetes.io/projected/58945644-f546-451e-bd8f-a3236df037f9-kube-api-access-cww6r\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7gdcm\" (UID: \"58945644-f546-451e-bd8f-a3236df037f9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7gdcm" Nov 01 20:01:03.064792 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:03.064772 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/58945644-f546-451e-bd8f-a3236df037f9-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7gdcm\" (UID: \"58945644-f546-451e-bd8f-a3236df037f9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7gdcm" Nov 01 20:01:03.064854 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:03.064840 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/58945644-f546-451e-bd8f-a3236df037f9-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7gdcm\" (UID: \"58945644-f546-451e-bd8f-a3236df037f9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7gdcm" Nov 01 20:01:03.079249 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:03.079222 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cww6r\" (UniqueName: \"kubernetes.io/projected/58945644-f546-451e-bd8f-a3236df037f9-kube-api-access-cww6r\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7gdcm\" (UID: \"58945644-f546-451e-bd8f-a3236df037f9\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7gdcm" Nov 01 20:01:03.150983 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:03.150955 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78v7xd" Nov 01 20:01:03.165224 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:03.165200 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvkpb\" (UniqueName: \"kubernetes.io/projected/aafb7b29-b974-46fa-9871-c9fbc188bb5f-kube-api-access-lvkpb\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lnfpz\" (UID: \"aafb7b29-b974-46fa-9871-c9fbc188bb5f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lnfpz" Nov 01 20:01:03.165338 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:03.165235 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aafb7b29-b974-46fa-9871-c9fbc188bb5f-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lnfpz\" (UID: \"aafb7b29-b974-46fa-9871-c9fbc188bb5f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lnfpz" Nov 01 20:01:03.165338 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:03.165254 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aafb7b29-b974-46fa-9871-c9fbc188bb5f-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lnfpz\" (UID: \"aafb7b29-b974-46fa-9871-c9fbc188bb5f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lnfpz" Nov 01 20:01:03.165615 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:03.165597 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aafb7b29-b974-46fa-9871-c9fbc188bb5f-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lnfpz\" (UID: \"aafb7b29-b974-46fa-9871-c9fbc188bb5f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lnfpz" Nov 01 20:01:03.165674 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:03.165651 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aafb7b29-b974-46fa-9871-c9fbc188bb5f-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lnfpz\" (UID: \"aafb7b29-b974-46fa-9871-c9fbc188bb5f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lnfpz" Nov 01 20:01:03.173344 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:03.173326 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvkpb\" (UniqueName: \"kubernetes.io/projected/aafb7b29-b974-46fa-9871-c9fbc188bb5f-kube-api-access-lvkpb\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lnfpz\" (UID: \"aafb7b29-b974-46fa-9871-c9fbc188bb5f\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lnfpz" Nov 01 20:01:03.250999 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:03.250974 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7gdcm" Nov 01 20:01:03.321631 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:03.321607 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atdgc9"] Nov 01 20:01:03.325571 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:01:03.325546 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2700c53f_9630_4fba_b2c3_ee481ecd5062.slice/crio-65ee1d1e570101064f22181dff42df831e81015b43ef115756382fbbe281e0c6 WatchSource:0}: Error finding container 65ee1d1e570101064f22181dff42df831e81015b43ef115756382fbbe281e0c6: Status 404 returned error can't find the container with id 65ee1d1e570101064f22181dff42df831e81015b43ef115756382fbbe281e0c6 Nov 01 20:01:03.327516 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:03.327499 5935 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 20:01:03.349890 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:03.349872 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lnfpz" Nov 01 20:01:03.368575 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:03.368552 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmkv4z"] Nov 01 20:01:03.371903 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:03.371885 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmkv4z" Nov 01 20:01:03.375109 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:03.375090 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-85k4s\"" Nov 01 20:01:03.379738 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:03.379720 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmkv4z"] Nov 01 20:01:03.424463 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:03.424429 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78v7xd"] Nov 01 20:01:03.428389 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:01:03.428359 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf4f598f_731b_4c35_85b7_86f0e700e546.slice/crio-dcc9cef2f33773331bb924d7e1cfb16d884f47654f148ebe4c01a7f61eeae2a3 WatchSource:0}: Error finding container dcc9cef2f33773331bb924d7e1cfb16d884f47654f148ebe4c01a7f61eeae2a3: Status 404 returned error can't find the container with id dcc9cef2f33773331bb924d7e1cfb16d884f47654f148ebe4c01a7f61eeae2a3 Nov 01 20:01:03.468247 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:03.468223 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mssmx\" (UniqueName: \"kubernetes.io/projected/1a40191e-dab8-4372-939f-ee7da2f9daa1-kube-api-access-mssmx\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmkv4z\" (UID: \"1a40191e-dab8-4372-939f-ee7da2f9daa1\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmkv4z" Nov 01 20:01:03.468359 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:03.468256 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1a40191e-dab8-4372-939f-ee7da2f9daa1-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmkv4z\" (UID: \"1a40191e-dab8-4372-939f-ee7da2f9daa1\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmkv4z" Nov 01 20:01:03.468359 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:03.468281 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1a40191e-dab8-4372-939f-ee7da2f9daa1-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmkv4z\" (UID: \"1a40191e-dab8-4372-939f-ee7da2f9daa1\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmkv4z" Nov 01 20:01:03.540740 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:03.540708 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7gdcm"] Nov 01 20:01:03.543241 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:01:03.543217 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58945644_f546_451e_bd8f_a3236df037f9.slice/crio-8856a2115b1dfa6a85ff2afdaa5c790b0fc6bed1bc457829b9311b100725c42f WatchSource:0}: Error finding container 8856a2115b1dfa6a85ff2afdaa5c790b0fc6bed1bc457829b9311b100725c42f: Status 404 returned error can't find the container with id 8856a2115b1dfa6a85ff2afdaa5c790b0fc6bed1bc457829b9311b100725c42f Nov 01 20:01:03.568959 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:03.568933 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mssmx\" (UniqueName: \"kubernetes.io/projected/1a40191e-dab8-4372-939f-ee7da2f9daa1-kube-api-access-mssmx\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmkv4z\" (UID: \"1a40191e-dab8-4372-939f-ee7da2f9daa1\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmkv4z" Nov 01 20:01:03.569042 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:03.568966 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1a40191e-dab8-4372-939f-ee7da2f9daa1-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmkv4z\" (UID: \"1a40191e-dab8-4372-939f-ee7da2f9daa1\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmkv4z" Nov 01 20:01:03.569042 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:03.568997 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1a40191e-dab8-4372-939f-ee7da2f9daa1-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmkv4z\" (UID: \"1a40191e-dab8-4372-939f-ee7da2f9daa1\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmkv4z" Nov 01 20:01:03.569396 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:03.569377 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1a40191e-dab8-4372-939f-ee7da2f9daa1-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmkv4z\" (UID: \"1a40191e-dab8-4372-939f-ee7da2f9daa1\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmkv4z" Nov 01 20:01:03.569439 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:03.569394 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1a40191e-dab8-4372-939f-ee7da2f9daa1-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmkv4z\" (UID: \"1a40191e-dab8-4372-939f-ee7da2f9daa1\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmkv4z" Nov 01 20:01:03.576468 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:03.576450 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mssmx\" (UniqueName: \"kubernetes.io/projected/1a40191e-dab8-4372-939f-ee7da2f9daa1-kube-api-access-mssmx\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmkv4z\" (UID: \"1a40191e-dab8-4372-939f-ee7da2f9daa1\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmkv4z" Nov 01 20:01:03.623012 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:03.622990 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lnfpz"] Nov 01 20:01:03.626242 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:01:03.626217 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaafb7b29_b974_46fa_9871_c9fbc188bb5f.slice/crio-8fa60b2016464ba4eb67220f1e6f2a7f41fe418cb993044399b2f9c038856034 WatchSource:0}: Error finding container 8fa60b2016464ba4eb67220f1e6f2a7f41fe418cb993044399b2f9c038856034: Status 404 returned error can't find the container with id 8fa60b2016464ba4eb67220f1e6f2a7f41fe418cb993044399b2f9c038856034 Nov 01 20:01:03.692556 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:03.692523 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmkv4z" Nov 01 20:01:03.793353 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:03.793317 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lnfpz" event={"ID":"aafb7b29-b974-46fa-9871-c9fbc188bb5f","Type":"ContainerStarted","Data":"8fa60b2016464ba4eb67220f1e6f2a7f41fe418cb993044399b2f9c038856034"} Nov 01 20:01:03.794678 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:03.794655 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78v7xd" event={"ID":"cf4f598f-731b-4c35-85b7-86f0e700e546","Type":"ContainerStarted","Data":"dcc9cef2f33773331bb924d7e1cfb16d884f47654f148ebe4c01a7f61eeae2a3"} Nov 01 20:01:03.796055 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:03.795989 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7gdcm" event={"ID":"58945644-f546-451e-bd8f-a3236df037f9","Type":"ContainerStarted","Data":"8856a2115b1dfa6a85ff2afdaa5c790b0fc6bed1bc457829b9311b100725c42f"} Nov 01 20:01:03.797319 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:03.797298 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atdgc9" event={"ID":"2700c53f-9630-4fba-b2c3-ee481ecd5062","Type":"ContainerStarted","Data":"65ee1d1e570101064f22181dff42df831e81015b43ef115756382fbbe281e0c6"} Nov 01 20:01:03.962748 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:03.962717 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmkv4z"] Nov 01 20:01:03.966747 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:01:03.966712 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a40191e_dab8_4372_939f_ee7da2f9daa1.slice/crio-41128a07d4c1681f35b5d28dc628fe1fbca235250f6392bc0420fcd6ead508cd WatchSource:0}: Error finding container 41128a07d4c1681f35b5d28dc628fe1fbca235250f6392bc0420fcd6ead508cd: Status 404 returned error can't find the container with id 41128a07d4c1681f35b5d28dc628fe1fbca235250f6392bc0420fcd6ead508cd Nov 01 20:01:04.804811 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:04.804731 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmkv4z" event={"ID":"1a40191e-dab8-4372-939f-ee7da2f9daa1","Type":"ContainerStarted","Data":"41128a07d4c1681f35b5d28dc628fe1fbca235250f6392bc0420fcd6ead508cd"} Nov 01 20:01:04.940573 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:04.940536 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gtmn2"] Nov 01 20:01:04.944939 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:04.944914 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gtmn2" Nov 01 20:01:04.954093 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:04.954066 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-t6zhp\"" Nov 01 20:01:04.954093 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:04.954078 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gtmn2"] Nov 01 20:01:04.980088 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:04.979729 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3667817b-d861-432a-8271-41f616b03cdc-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gtmn2\" (UID: \"3667817b-d861-432a-8271-41f616b03cdc\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gtmn2" Nov 01 20:01:04.980088 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:04.979778 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnhdg\" (UniqueName: \"kubernetes.io/projected/3667817b-d861-432a-8271-41f616b03cdc-kube-api-access-pnhdg\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gtmn2\" (UID: \"3667817b-d861-432a-8271-41f616b03cdc\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gtmn2" Nov 01 20:01:04.980088 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:04.979850 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3667817b-d861-432a-8271-41f616b03cdc-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gtmn2\" (UID: \"3667817b-d861-432a-8271-41f616b03cdc\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gtmn2" Nov 01 20:01:05.081679 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:05.081601 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3667817b-d861-432a-8271-41f616b03cdc-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gtmn2\" (UID: \"3667817b-d861-432a-8271-41f616b03cdc\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gtmn2" Nov 01 20:01:05.081679 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:05.081641 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnhdg\" (UniqueName: \"kubernetes.io/projected/3667817b-d861-432a-8271-41f616b03cdc-kube-api-access-pnhdg\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gtmn2\" (UID: \"3667817b-d861-432a-8271-41f616b03cdc\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gtmn2" Nov 01 20:01:05.082037 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:05.081713 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3667817b-d861-432a-8271-41f616b03cdc-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gtmn2\" (UID: \"3667817b-d861-432a-8271-41f616b03cdc\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gtmn2" Nov 01 20:01:05.082117 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:05.082044 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3667817b-d861-432a-8271-41f616b03cdc-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gtmn2\" (UID: \"3667817b-d861-432a-8271-41f616b03cdc\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gtmn2" Nov 01 20:01:05.082117 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:05.082089 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3667817b-d861-432a-8271-41f616b03cdc-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gtmn2\" (UID: \"3667817b-d861-432a-8271-41f616b03cdc\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gtmn2" Nov 01 20:01:05.110139 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:05.110083 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnhdg\" (UniqueName: \"kubernetes.io/projected/3667817b-d861-432a-8271-41f616b03cdc-kube-api-access-pnhdg\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gtmn2\" (UID: \"3667817b-d861-432a-8271-41f616b03cdc\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gtmn2" Nov 01 20:01:05.271658 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:05.271623 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gtmn2" Nov 01 20:01:05.583222 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:05.583134 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gtmn2"] Nov 01 20:01:05.608791 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:01:05.608755 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3667817b_d861_432a_8271_41f616b03cdc.slice/crio-3c8b13b45bed3141e0a9682eb1c2aaf469e6311ac83111ff15109000ffcf675c WatchSource:0}: Error finding container 3c8b13b45bed3141e0a9682eb1c2aaf469e6311ac83111ff15109000ffcf675c: Status 404 returned error can't find the container with id 3c8b13b45bed3141e0a9682eb1c2aaf469e6311ac83111ff15109000ffcf675c Nov 01 20:01:05.811503 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:05.811421 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gtmn2" event={"ID":"3667817b-d861-432a-8271-41f616b03cdc","Type":"ContainerStarted","Data":"3c8b13b45bed3141e0a9682eb1c2aaf469e6311ac83111ff15109000ffcf675c"} Nov 01 20:01:05.878514 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:01:05.878390 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:01:07.315193 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:07.313432 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-ljtcq"] Nov 01 20:01:07.317423 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:07.317396 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-ljtcq" Nov 01 20:01:07.321679 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:07.321592 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-njf76\"" Nov 01 20:01:07.322437 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:07.322042 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 01 20:01:07.322437 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:07.322086 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 01 20:01:07.322437 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:07.322138 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 01 20:01:07.344243 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:07.344211 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-ljtcq"] Nov 01 20:01:07.406123 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:07.405568 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03014616-4520-48bb-a548-66485577e26c-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-ljtcq\" (UID: \"03014616-4520-48bb-a548-66485577e26c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-ljtcq" Nov 01 20:01:07.406123 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:07.405637 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03014616-4520-48bb-a548-66485577e26c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-ljtcq\" (UID: \"03014616-4520-48bb-a548-66485577e26c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-ljtcq" Nov 01 20:01:07.406123 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:07.405669 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8br9z\" (UniqueName: \"kubernetes.io/projected/03014616-4520-48bb-a548-66485577e26c-kube-api-access-8br9z\") pod \"maas-default-gateway-openshift-default-fd4b99879-ljtcq\" (UID: \"03014616-4520-48bb-a548-66485577e26c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-ljtcq" Nov 01 20:01:07.406123 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:07.405706 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03014616-4520-48bb-a548-66485577e26c-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-ljtcq\" (UID: \"03014616-4520-48bb-a548-66485577e26c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-ljtcq" Nov 01 20:01:07.406123 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:07.405759 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03014616-4520-48bb-a548-66485577e26c-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-ljtcq\" (UID: \"03014616-4520-48bb-a548-66485577e26c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-ljtcq" Nov 01 20:01:07.406123 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:07.405788 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03014616-4520-48bb-a548-66485577e26c-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-ljtcq\" (UID: \"03014616-4520-48bb-a548-66485577e26c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-ljtcq" Nov 01 20:01:07.406123 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:07.405815 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03014616-4520-48bb-a548-66485577e26c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-ljtcq\" (UID: \"03014616-4520-48bb-a548-66485577e26c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-ljtcq" Nov 01 20:01:07.406123 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:07.405853 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03014616-4520-48bb-a548-66485577e26c-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-ljtcq\" (UID: \"03014616-4520-48bb-a548-66485577e26c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-ljtcq" Nov 01 20:01:07.406123 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:07.405891 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03014616-4520-48bb-a548-66485577e26c-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-ljtcq\" (UID: \"03014616-4520-48bb-a548-66485577e26c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-ljtcq" Nov 01 20:01:07.509784 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:07.507206 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03014616-4520-48bb-a548-66485577e26c-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-ljtcq\" (UID: \"03014616-4520-48bb-a548-66485577e26c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-ljtcq" Nov 01 20:01:07.509784 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:07.507269 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03014616-4520-48bb-a548-66485577e26c-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-ljtcq\" (UID: \"03014616-4520-48bb-a548-66485577e26c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-ljtcq" Nov 01 20:01:07.509784 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:07.507335 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03014616-4520-48bb-a548-66485577e26c-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-ljtcq\" (UID: \"03014616-4520-48bb-a548-66485577e26c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-ljtcq" Nov 01 20:01:07.509784 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:07.507385 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03014616-4520-48bb-a548-66485577e26c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-ljtcq\" (UID: \"03014616-4520-48bb-a548-66485577e26c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-ljtcq" Nov 01 20:01:07.509784 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:07.507410 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8br9z\" (UniqueName: \"kubernetes.io/projected/03014616-4520-48bb-a548-66485577e26c-kube-api-access-8br9z\") pod \"maas-default-gateway-openshift-default-fd4b99879-ljtcq\" (UID: \"03014616-4520-48bb-a548-66485577e26c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-ljtcq" Nov 01 20:01:07.509784 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:07.507448 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03014616-4520-48bb-a548-66485577e26c-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-ljtcq\" (UID: \"03014616-4520-48bb-a548-66485577e26c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-ljtcq" Nov 01 20:01:07.509784 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:07.507478 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03014616-4520-48bb-a548-66485577e26c-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-ljtcq\" (UID: \"03014616-4520-48bb-a548-66485577e26c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-ljtcq" Nov 01 20:01:07.509784 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:07.507505 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03014616-4520-48bb-a548-66485577e26c-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-ljtcq\" (UID: \"03014616-4520-48bb-a548-66485577e26c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-ljtcq" Nov 01 20:01:07.509784 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:07.507531 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03014616-4520-48bb-a548-66485577e26c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-ljtcq\" (UID: \"03014616-4520-48bb-a548-66485577e26c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-ljtcq" Nov 01 20:01:07.509784 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:07.508265 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03014616-4520-48bb-a548-66485577e26c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-ljtcq\" (UID: \"03014616-4520-48bb-a548-66485577e26c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-ljtcq" Nov 01 20:01:07.509784 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:07.508561 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03014616-4520-48bb-a548-66485577e26c-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-ljtcq\" (UID: \"03014616-4520-48bb-a548-66485577e26c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-ljtcq" Nov 01 20:01:07.509784 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:07.509079 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03014616-4520-48bb-a548-66485577e26c-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-ljtcq\" (UID: \"03014616-4520-48bb-a548-66485577e26c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-ljtcq" Nov 01 20:01:07.510536 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:07.510000 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03014616-4520-48bb-a548-66485577e26c-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-ljtcq\" (UID: \"03014616-4520-48bb-a548-66485577e26c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-ljtcq" Nov 01 20:01:07.510536 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:07.510235 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03014616-4520-48bb-a548-66485577e26c-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-ljtcq\" (UID: \"03014616-4520-48bb-a548-66485577e26c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-ljtcq" Nov 01 20:01:07.514053 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:07.514028 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03014616-4520-48bb-a548-66485577e26c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-ljtcq\" (UID: \"03014616-4520-48bb-a548-66485577e26c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-ljtcq" Nov 01 20:01:07.516928 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:07.516852 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03014616-4520-48bb-a548-66485577e26c-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-ljtcq\" (UID: \"03014616-4520-48bb-a548-66485577e26c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-ljtcq" Nov 01 20:01:07.520265 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:07.520240 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8br9z\" (UniqueName: \"kubernetes.io/projected/03014616-4520-48bb-a548-66485577e26c-kube-api-access-8br9z\") pod \"maas-default-gateway-openshift-default-fd4b99879-ljtcq\" (UID: \"03014616-4520-48bb-a548-66485577e26c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-ljtcq" Nov 01 20:01:07.522776 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:07.522712 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03014616-4520-48bb-a548-66485577e26c-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-ljtcq\" (UID: \"03014616-4520-48bb-a548-66485577e26c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-ljtcq" Nov 01 20:01:07.657803 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:07.657513 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-ljtcq" Nov 01 20:01:08.011779 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:08.011746 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-ljtcq"] Nov 01 20:01:08.019240 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:01:08.019204 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03014616_4520_48bb_a548_66485577e26c.slice/crio-64199ed5357b99d5e20ada9dfb111402ad48ef4914c8e6049d2ea05d27ee021f WatchSource:0}: Error finding container 64199ed5357b99d5e20ada9dfb111402ad48ef4914c8e6049d2ea05d27ee021f: Status 404 returned error can't find the container with id 64199ed5357b99d5e20ada9dfb111402ad48ef4914c8e6049d2ea05d27ee021f Nov 01 20:01:08.833613 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:08.833527 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-ljtcq" event={"ID":"03014616-4520-48bb-a548-66485577e26c","Type":"ContainerStarted","Data":"64199ed5357b99d5e20ada9dfb111402ad48ef4914c8e6049d2ea05d27ee021f"} Nov 01 20:01:09.898119 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:09.898084 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ff4lq"] Nov 01 20:01:09.905517 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:09.904890 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ff4lq" Nov 01 20:01:09.914221 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:09.914165 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ff4lq"] Nov 01 20:01:09.931378 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:09.931352 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qnzm\" (UniqueName: \"kubernetes.io/projected/42551e2c-2573-4983-8860-f87e21da11c5-kube-api-access-5qnzm\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ff4lq\" (UID: \"42551e2c-2573-4983-8860-f87e21da11c5\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ff4lq" Nov 01 20:01:09.931508 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:09.931390 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/42551e2c-2573-4983-8860-f87e21da11c5-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ff4lq\" (UID: \"42551e2c-2573-4983-8860-f87e21da11c5\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ff4lq" Nov 01 20:01:09.931508 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:09.931433 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/42551e2c-2573-4983-8860-f87e21da11c5-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ff4lq\" (UID: \"42551e2c-2573-4983-8860-f87e21da11c5\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ff4lq" Nov 01 20:01:10.032351 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:10.032313 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qnzm\" (UniqueName: \"kubernetes.io/projected/42551e2c-2573-4983-8860-f87e21da11c5-kube-api-access-5qnzm\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ff4lq\" (UID: \"42551e2c-2573-4983-8860-f87e21da11c5\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ff4lq" Nov 01 20:01:10.032496 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:10.032370 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/42551e2c-2573-4983-8860-f87e21da11c5-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ff4lq\" (UID: \"42551e2c-2573-4983-8860-f87e21da11c5\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ff4lq" Nov 01 20:01:10.032496 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:10.032419 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/42551e2c-2573-4983-8860-f87e21da11c5-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ff4lq\" (UID: \"42551e2c-2573-4983-8860-f87e21da11c5\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ff4lq" Nov 01 20:01:10.032942 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:10.032913 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/42551e2c-2573-4983-8860-f87e21da11c5-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ff4lq\" (UID: \"42551e2c-2573-4983-8860-f87e21da11c5\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ff4lq" Nov 01 20:01:10.033551 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:10.033500 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/42551e2c-2573-4983-8860-f87e21da11c5-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ff4lq\" (UID: \"42551e2c-2573-4983-8860-f87e21da11c5\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ff4lq" Nov 01 20:01:10.054380 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:10.054351 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qnzm\" (UniqueName: \"kubernetes.io/projected/42551e2c-2573-4983-8860-f87e21da11c5-kube-api-access-5qnzm\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ff4lq\" (UID: \"42551e2c-2573-4983-8860-f87e21da11c5\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ff4lq" Nov 01 20:01:10.234496 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:10.233716 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ff4lq" Nov 01 20:01:10.592372 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:10.592095 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ff4lq"] Nov 01 20:01:10.604383 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:01:10.604339 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42551e2c_2573_4983_8860_f87e21da11c5.slice/crio-a8d0f6c259d619edcc2fc083cda48e7aa7b0b6778555c27b24cb6d067563ceae WatchSource:0}: Error finding container a8d0f6c259d619edcc2fc083cda48e7aa7b0b6778555c27b24cb6d067563ceae: Status 404 returned error can't find the container with id a8d0f6c259d619edcc2fc083cda48e7aa7b0b6778555c27b24cb6d067563ceae Nov 01 20:01:10.851388 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:10.851303 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ff4lq" event={"ID":"42551e2c-2573-4983-8860-f87e21da11c5","Type":"ContainerStarted","Data":"a8d0f6c259d619edcc2fc083cda48e7aa7b0b6778555c27b24cb6d067563ceae"} Nov 01 20:01:16.709931 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:16.709898 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:01:16.710329 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:16.709979 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:01:16.710329 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:16.710024 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:01:16.904899 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:16.904870 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atdgc9" event={"ID":"2700c53f-9630-4fba-b2c3-ee481ecd5062","Type":"ContainerStarted","Data":"892fffef24abffa3d384862ddcad23e06a85caffdeab355a99d3bc13125e3b5f"} Nov 01 20:01:16.911125 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:16.911068 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-ljtcq" event={"ID":"03014616-4520-48bb-a548-66485577e26c","Type":"ContainerStarted","Data":"76c8578d02abb6f5d2c0a60721d1fcc733fbed26f7bfb19d7efa857a4c1a5df4"} Nov 01 20:01:16.942322 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:16.942251 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-ljtcq" podStartSLOduration=1.258139566 podStartE2EDuration="9.942236324s" totalImagesPullingTime="8.684096758s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:01:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:01:16.940067402 +0000 UTC m=+122.713227914" watchObservedRunningTime="2026-11-01 20:01:16.942236324 +0000 UTC m=+122.715396830" Nov 01 20:01:17.660225 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:17.660186 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-ljtcq" Nov 01 20:01:17.666221 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:17.666195 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-ljtcq" Nov 01 20:01:17.917766 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:17.917430 5935 generic.go:356] "Generic (PLEG): container finished" podID="aafb7b29-b974-46fa-9871-c9fbc188bb5f" containerID="8ff2e8d070a65c2053e6f2b2001de254d35613322f5a4f4378ab2acbb6d0ea77" exitCode=0 Nov 01 20:01:17.917766 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:17.917533 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lnfpz" event={"ID":"aafb7b29-b974-46fa-9871-c9fbc188bb5f","Type":"ContainerDied","Data":"8ff2e8d070a65c2053e6f2b2001de254d35613322f5a4f4378ab2acbb6d0ea77"} Nov 01 20:01:17.921396 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:17.921368 5935 generic.go:356] "Generic (PLEG): container finished" podID="58945644-f546-451e-bd8f-a3236df037f9" containerID="9600e89b9ba08341b585aadbf7a2654c1826edef344100bd6f6539e16c6f38d8" exitCode=0 Nov 01 20:01:17.921501 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:17.921482 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7gdcm" event={"ID":"58945644-f546-451e-bd8f-a3236df037f9","Type":"ContainerDied","Data":"9600e89b9ba08341b585aadbf7a2654c1826edef344100bd6f6539e16c6f38d8"} Nov 01 20:01:17.924173 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:17.924148 5935 generic.go:356] "Generic (PLEG): container finished" podID="2700c53f-9630-4fba-b2c3-ee481ecd5062" containerID="892fffef24abffa3d384862ddcad23e06a85caffdeab355a99d3bc13125e3b5f" exitCode=0 Nov 01 20:01:17.924262 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:17.924228 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atdgc9" event={"ID":"2700c53f-9630-4fba-b2c3-ee481ecd5062","Type":"ContainerDied","Data":"892fffef24abffa3d384862ddcad23e06a85caffdeab355a99d3bc13125e3b5f"} Nov 01 20:01:17.924735 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:17.924578 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-ljtcq" Nov 01 20:01:17.925618 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:17.925599 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-ljtcq" Nov 01 20:01:18.929479 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:18.929409 5935 generic.go:356] "Generic (PLEG): container finished" podID="42551e2c-2573-4983-8860-f87e21da11c5" containerID="48934a1db7d99c056651eb096178bf03884eb5bbc823034c002f90038148a7c5" exitCode=0 Nov 01 20:01:18.929906 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:18.929477 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ff4lq" event={"ID":"42551e2c-2573-4983-8860-f87e21da11c5","Type":"ContainerDied","Data":"48934a1db7d99c056651eb096178bf03884eb5bbc823034c002f90038148a7c5"} Nov 01 20:01:18.931084 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:18.931052 5935 generic.go:356] "Generic (PLEG): container finished" podID="cf4f598f-731b-4c35-85b7-86f0e700e546" containerID="2749c89f723d2d98b27868ef5479696b3ae680ca9bb6c7b02953986e99a746c0" exitCode=0 Nov 01 20:01:18.931166 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:18.931121 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78v7xd" event={"ID":"cf4f598f-731b-4c35-85b7-86f0e700e546","Type":"ContainerDied","Data":"2749c89f723d2d98b27868ef5479696b3ae680ca9bb6c7b02953986e99a746c0"} Nov 01 20:01:18.932608 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:18.932586 5935 generic.go:356] "Generic (PLEG): container finished" podID="1a40191e-dab8-4372-939f-ee7da2f9daa1" containerID="c1c6aeebe589ea1b1de39d5fedd9a1b5c69f2d4aea8e37804b692d02f8404c92" exitCode=0 Nov 01 20:01:18.932693 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:18.932629 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmkv4z" event={"ID":"1a40191e-dab8-4372-939f-ee7da2f9daa1","Type":"ContainerDied","Data":"c1c6aeebe589ea1b1de39d5fedd9a1b5c69f2d4aea8e37804b692d02f8404c92"} Nov 01 20:01:19.938310 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:19.938264 5935 generic.go:356] "Generic (PLEG): container finished" podID="aafb7b29-b974-46fa-9871-c9fbc188bb5f" containerID="c8491886fb886387ad92ae2565925f612c717ed320c15a908d3c204af6ff69a3" exitCode=0 Nov 01 20:01:19.938716 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:19.938318 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lnfpz" event={"ID":"aafb7b29-b974-46fa-9871-c9fbc188bb5f","Type":"ContainerDied","Data":"c8491886fb886387ad92ae2565925f612c717ed320c15a908d3c204af6ff69a3"} Nov 01 20:01:19.940325 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:19.940301 5935 generic.go:356] "Generic (PLEG): container finished" podID="58945644-f546-451e-bd8f-a3236df037f9" containerID="48a13f64fc18c1c40b08b5c5307c460ec77de0e356f395f43b12442474a2108f" exitCode=0 Nov 01 20:01:19.940401 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:19.940378 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7gdcm" event={"ID":"58945644-f546-451e-bd8f-a3236df037f9","Type":"ContainerDied","Data":"48a13f64fc18c1c40b08b5c5307c460ec77de0e356f395f43b12442474a2108f"} Nov 01 20:01:19.942846 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:19.942827 5935 generic.go:356] "Generic (PLEG): container finished" podID="3667817b-d861-432a-8271-41f616b03cdc" containerID="edd4a423b8e19d924f24632f0306865d8800fee69db7cd0edd05408b8cd0da48" exitCode=0 Nov 01 20:01:19.942936 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:19.942915 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gtmn2" event={"ID":"3667817b-d861-432a-8271-41f616b03cdc","Type":"ContainerDied","Data":"edd4a423b8e19d924f24632f0306865d8800fee69db7cd0edd05408b8cd0da48"} Nov 01 20:01:19.945084 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:19.944928 5935 generic.go:356] "Generic (PLEG): container finished" podID="2700c53f-9630-4fba-b2c3-ee481ecd5062" containerID="7daebe772ab2380289cf7c3b6f33de38b2c663419b08a5b75619a93ff1e900b8" exitCode=0 Nov 01 20:01:19.945084 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:19.945004 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atdgc9" event={"ID":"2700c53f-9630-4fba-b2c3-ee481ecd5062","Type":"ContainerDied","Data":"7daebe772ab2380289cf7c3b6f33de38b2c663419b08a5b75619a93ff1e900b8"} Nov 01 20:01:20.953212 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:20.953040 5935 generic.go:356] "Generic (PLEG): container finished" podID="42551e2c-2573-4983-8860-f87e21da11c5" containerID="7f76ccfc7b6cf3899af13cbd903e8b24ca850b716a5f5783dd1797e8bc88ff55" exitCode=0 Nov 01 20:01:20.953212 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:20.953137 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ff4lq" event={"ID":"42551e2c-2573-4983-8860-f87e21da11c5","Type":"ContainerDied","Data":"7f76ccfc7b6cf3899af13cbd903e8b24ca850b716a5f5783dd1797e8bc88ff55"} Nov 01 20:01:20.957450 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:20.957425 5935 generic.go:356] "Generic (PLEG): container finished" podID="cf4f598f-731b-4c35-85b7-86f0e700e546" containerID="c82ced606e33164614d4a8597b14e233f5d1dcb4feadf9071456b06643713105" exitCode=0 Nov 01 20:01:20.957565 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:20.957465 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78v7xd" event={"ID":"cf4f598f-731b-4c35-85b7-86f0e700e546","Type":"ContainerDied","Data":"c82ced606e33164614d4a8597b14e233f5d1dcb4feadf9071456b06643713105"} Nov 01 20:01:23.987840 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:23.987803 5935 generic.go:356] "Generic (PLEG): container finished" podID="1a40191e-dab8-4372-939f-ee7da2f9daa1" containerID="c26817e51de53dd03815708c6345500234b1cf70019625000b389bdae64e4649" exitCode=0 Nov 01 20:01:23.988326 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:23.987862 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmkv4z" event={"ID":"1a40191e-dab8-4372-939f-ee7da2f9daa1","Type":"ContainerDied","Data":"c26817e51de53dd03815708c6345500234b1cf70019625000b389bdae64e4649"} Nov 01 20:01:24.997687 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:24.997645 5935 generic.go:356] "Generic (PLEG): container finished" podID="3667817b-d861-432a-8271-41f616b03cdc" containerID="b32b0876649000b4f45d9d4f8fbad697dc72d5585d5de365461c3dd9e68270d7" exitCode=0 Nov 01 20:01:24.998145 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:24.997761 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gtmn2" event={"ID":"3667817b-d861-432a-8271-41f616b03cdc","Type":"ContainerDied","Data":"b32b0876649000b4f45d9d4f8fbad697dc72d5585d5de365461c3dd9e68270d7"} Nov 01 20:01:34.654213 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:34.652609 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-lvm4r"] Nov 01 20:01:34.671644 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:34.671604 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-lvm4r" Nov 01 20:01:34.701264 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:34.701025 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-lvm4r"] Nov 01 20:01:34.740073 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:34.740025 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 01 20:01:34.740073 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:34.740029 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 01 20:01:34.746198 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:34.746177 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 01 20:01:34.746328 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:34.746279 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 01 20:01:34.746430 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:34.746393 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 01 20:01:34.746483 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:34.746450 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-rw9ss\"" Nov 01 20:01:34.784048 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:34.783622 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/48a12663-97c4-486c-9462-183947bbfdc1-tmp\") pod \"opendatahub-operator-controller-manager-689bcf9f9-lvm4r\" (UID: \"48a12663-97c4-486c-9462-183947bbfdc1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-lvm4r" Nov 01 20:01:34.784048 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:34.783662 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/48a12663-97c4-486c-9462-183947bbfdc1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-689bcf9f9-lvm4r\" (UID: \"48a12663-97c4-486c-9462-183947bbfdc1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-lvm4r" Nov 01 20:01:34.784048 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:34.783708 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/48a12663-97c4-486c-9462-183947bbfdc1-webhook-cert\") pod \"opendatahub-operator-controller-manager-689bcf9f9-lvm4r\" (UID: \"48a12663-97c4-486c-9462-183947bbfdc1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-lvm4r" Nov 01 20:01:34.784048 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:34.783748 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kbfl\" (UniqueName: \"kubernetes.io/projected/48a12663-97c4-486c-9462-183947bbfdc1-kube-api-access-7kbfl\") pod \"opendatahub-operator-controller-manager-689bcf9f9-lvm4r\" (UID: \"48a12663-97c4-486c-9462-183947bbfdc1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-lvm4r" Nov 01 20:01:34.784048 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:34.783804 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/48a12663-97c4-486c-9462-183947bbfdc1-metrics-cert\") pod \"opendatahub-operator-controller-manager-689bcf9f9-lvm4r\" (UID: \"48a12663-97c4-486c-9462-183947bbfdc1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-lvm4r" Nov 01 20:01:34.784048 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:34.783889 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/48a12663-97c4-486c-9462-183947bbfdc1-manifests\") pod \"opendatahub-operator-controller-manager-689bcf9f9-lvm4r\" (UID: \"48a12663-97c4-486c-9462-183947bbfdc1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-lvm4r" Nov 01 20:01:34.891704 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:34.891026 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/48a12663-97c4-486c-9462-183947bbfdc1-webhook-cert\") pod \"opendatahub-operator-controller-manager-689bcf9f9-lvm4r\" (UID: \"48a12663-97c4-486c-9462-183947bbfdc1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-lvm4r" Nov 01 20:01:34.891704 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:34.891090 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kbfl\" (UniqueName: \"kubernetes.io/projected/48a12663-97c4-486c-9462-183947bbfdc1-kube-api-access-7kbfl\") pod \"opendatahub-operator-controller-manager-689bcf9f9-lvm4r\" (UID: \"48a12663-97c4-486c-9462-183947bbfdc1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-lvm4r" Nov 01 20:01:34.891704 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:34.891149 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/48a12663-97c4-486c-9462-183947bbfdc1-metrics-cert\") pod \"opendatahub-operator-controller-manager-689bcf9f9-lvm4r\" (UID: \"48a12663-97c4-486c-9462-183947bbfdc1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-lvm4r" Nov 01 20:01:34.891704 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:34.891228 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/48a12663-97c4-486c-9462-183947bbfdc1-manifests\") pod \"opendatahub-operator-controller-manager-689bcf9f9-lvm4r\" (UID: \"48a12663-97c4-486c-9462-183947bbfdc1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-lvm4r" Nov 01 20:01:34.891704 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:34.891258 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/48a12663-97c4-486c-9462-183947bbfdc1-tmp\") pod \"opendatahub-operator-controller-manager-689bcf9f9-lvm4r\" (UID: \"48a12663-97c4-486c-9462-183947bbfdc1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-lvm4r" Nov 01 20:01:34.891704 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:34.891273 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/48a12663-97c4-486c-9462-183947bbfdc1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-689bcf9f9-lvm4r\" (UID: \"48a12663-97c4-486c-9462-183947bbfdc1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-lvm4r" Nov 01 20:01:34.893219 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:34.892930 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/48a12663-97c4-486c-9462-183947bbfdc1-manifests\") pod \"opendatahub-operator-controller-manager-689bcf9f9-lvm4r\" (UID: \"48a12663-97c4-486c-9462-183947bbfdc1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-lvm4r" Nov 01 20:01:34.896591 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:34.896541 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/48a12663-97c4-486c-9462-183947bbfdc1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-689bcf9f9-lvm4r\" (UID: \"48a12663-97c4-486c-9462-183947bbfdc1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-lvm4r" Nov 01 20:01:34.896854 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:34.896823 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/48a12663-97c4-486c-9462-183947bbfdc1-metrics-cert\") pod \"opendatahub-operator-controller-manager-689bcf9f9-lvm4r\" (UID: \"48a12663-97c4-486c-9462-183947bbfdc1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-lvm4r" Nov 01 20:01:34.896934 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:34.896903 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/48a12663-97c4-486c-9462-183947bbfdc1-webhook-cert\") pod \"opendatahub-operator-controller-manager-689bcf9f9-lvm4r\" (UID: \"48a12663-97c4-486c-9462-183947bbfdc1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-lvm4r" Nov 01 20:01:34.898448 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:34.898400 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/48a12663-97c4-486c-9462-183947bbfdc1-tmp\") pod \"opendatahub-operator-controller-manager-689bcf9f9-lvm4r\" (UID: \"48a12663-97c4-486c-9462-183947bbfdc1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-lvm4r" Nov 01 20:01:34.905271 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:34.904917 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kbfl\" (UniqueName: \"kubernetes.io/projected/48a12663-97c4-486c-9462-183947bbfdc1-kube-api-access-7kbfl\") pod \"opendatahub-operator-controller-manager-689bcf9f9-lvm4r\" (UID: \"48a12663-97c4-486c-9462-183947bbfdc1\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-lvm4r" Nov 01 20:01:35.077723 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:35.077577 5935 generic.go:356] "Generic (PLEG): container finished" podID="58945644-f546-451e-bd8f-a3236df037f9" containerID="3ba69713ae29a3e74d87ee792ed5252b8b664e039b9c00656f6fd4894ff87f4d" exitCode=0 Nov 01 20:01:35.077723 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:35.077617 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7gdcm" event={"ID":"58945644-f546-451e-bd8f-a3236df037f9","Type":"ContainerDied","Data":"3ba69713ae29a3e74d87ee792ed5252b8b664e039b9c00656f6fd4894ff87f4d"} Nov 01 20:01:35.080364 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:35.080184 5935 generic.go:356] "Generic (PLEG): container finished" podID="3667817b-d861-432a-8271-41f616b03cdc" containerID="33ae0789e8f4d892e7b8277978043d3a014f0998d6a35df03906800ce9972c71" exitCode=0 Nov 01 20:01:35.080364 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:35.080214 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gtmn2" event={"ID":"3667817b-d861-432a-8271-41f616b03cdc","Type":"ContainerDied","Data":"33ae0789e8f4d892e7b8277978043d3a014f0998d6a35df03906800ce9972c71"} Nov 01 20:01:35.083023 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:35.082988 5935 generic.go:356] "Generic (PLEG): container finished" podID="aafb7b29-b974-46fa-9871-c9fbc188bb5f" containerID="f26cca5c27e29345aae248172efb92bc4b3bd1ca894b68db888624b15e58b812" exitCode=0 Nov 01 20:01:35.083149 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:35.083043 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lnfpz" event={"ID":"aafb7b29-b974-46fa-9871-c9fbc188bb5f","Type":"ContainerDied","Data":"f26cca5c27e29345aae248172efb92bc4b3bd1ca894b68db888624b15e58b812"} Nov 01 20:01:35.122870 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:35.122842 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-lvm4r" Nov 01 20:01:35.443532 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:35.443502 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-lvm4r"] Nov 01 20:01:35.449658 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:01:35.449627 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48a12663_97c4_486c_9462_183947bbfdc1.slice/crio-4b9a67e2d1cd5c051df31f0bb875f64a25bd5ead75845575cc8f3df4f07a932d WatchSource:0}: Error finding container 4b9a67e2d1cd5c051df31f0bb875f64a25bd5ead75845575cc8f3df4f07a932d: Status 404 returned error can't find the container with id 4b9a67e2d1cd5c051df31f0bb875f64a25bd5ead75845575cc8f3df4f07a932d Nov 01 20:01:35.847471 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:01:35.847433 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:01:36.088915 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:36.088821 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-lvm4r" event={"ID":"48a12663-97c4-486c-9462-183947bbfdc1","Type":"ContainerStarted","Data":"4b9a67e2d1cd5c051df31f0bb875f64a25bd5ead75845575cc8f3df4f07a932d"} Nov 01 20:01:36.091777 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:36.091667 5935 generic.go:356] "Generic (PLEG): container finished" podID="2700c53f-9630-4fba-b2c3-ee481ecd5062" containerID="9fd6a1059a940455dec1d9791cb2f4d1c2d34dd762b79dd3cec248e8b54eb8ec" exitCode=0 Nov 01 20:01:36.091777 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:36.091749 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atdgc9" event={"ID":"2700c53f-9630-4fba-b2c3-ee481ecd5062","Type":"ContainerDied","Data":"9fd6a1059a940455dec1d9791cb2f4d1c2d34dd762b79dd3cec248e8b54eb8ec"} Nov 01 20:01:36.094512 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:36.094402 5935 generic.go:356] "Generic (PLEG): container finished" podID="42551e2c-2573-4983-8860-f87e21da11c5" containerID="6ad71386a6b49ceab5b158394968e50427ca1d4f578616f6aad02bb61fadbc4c" exitCode=0 Nov 01 20:01:36.094512 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:36.094486 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ff4lq" event={"ID":"42551e2c-2573-4983-8860-f87e21da11c5","Type":"ContainerDied","Data":"6ad71386a6b49ceab5b158394968e50427ca1d4f578616f6aad02bb61fadbc4c"} Nov 01 20:01:36.097307 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:36.097160 5935 generic.go:356] "Generic (PLEG): container finished" podID="cf4f598f-731b-4c35-85b7-86f0e700e546" containerID="593bfe449fb0813c537407747093f5d49582dc70614ef96b3b25f577e80ee61b" exitCode=0 Nov 01 20:01:36.097307 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:36.097307 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78v7xd" event={"ID":"cf4f598f-731b-4c35-85b7-86f0e700e546","Type":"ContainerDied","Data":"593bfe449fb0813c537407747093f5d49582dc70614ef96b3b25f577e80ee61b"} Nov 01 20:01:36.309241 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:36.309213 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lnfpz" Nov 01 20:01:36.346243 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:36.346147 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7gdcm" Nov 01 20:01:36.362105 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:36.362080 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gtmn2" Nov 01 20:01:36.406364 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:36.406321 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aafb7b29-b974-46fa-9871-c9fbc188bb5f-util\" (UniqueName: \"kubernetes.io/empty-dir/aafb7b29-b974-46fa-9871-c9fbc188bb5f-util\") pod \"aafb7b29-b974-46fa-9871-c9fbc188bb5f\" (UID: \"aafb7b29-b974-46fa-9871-c9fbc188bb5f\") " Nov 01 20:01:36.406506 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:36.406368 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aafb7b29-b974-46fa-9871-c9fbc188bb5f-kube-api-access-lvkpb\" (UniqueName: \"kubernetes.io/projected/aafb7b29-b974-46fa-9871-c9fbc188bb5f-kube-api-access-lvkpb\") pod \"aafb7b29-b974-46fa-9871-c9fbc188bb5f\" (UID: \"aafb7b29-b974-46fa-9871-c9fbc188bb5f\") " Nov 01 20:01:36.406506 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:36.406433 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aafb7b29-b974-46fa-9871-c9fbc188bb5f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/aafb7b29-b974-46fa-9871-c9fbc188bb5f-bundle\") pod \"aafb7b29-b974-46fa-9871-c9fbc188bb5f\" (UID: \"aafb7b29-b974-46fa-9871-c9fbc188bb5f\") " Nov 01 20:01:36.407168 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:36.407143 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aafb7b29-b974-46fa-9871-c9fbc188bb5f-bundle" pod "aafb7b29-b974-46fa-9871-c9fbc188bb5f" (UID: "aafb7b29-b974-46fa-9871-c9fbc188bb5f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:01:36.411468 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:36.411433 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aafb7b29-b974-46fa-9871-c9fbc188bb5f-kube-api-access-lvkpb" pod "aafb7b29-b974-46fa-9871-c9fbc188bb5f" (UID: "aafb7b29-b974-46fa-9871-c9fbc188bb5f"). InnerVolumeSpecName "kube-api-access-lvkpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:01:36.415113 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:36.415084 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aafb7b29-b974-46fa-9871-c9fbc188bb5f-util" pod "aafb7b29-b974-46fa-9871-c9fbc188bb5f" (UID: "aafb7b29-b974-46fa-9871-c9fbc188bb5f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:01:36.507604 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:36.507580 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3667817b-d861-432a-8271-41f616b03cdc-bundle\" (UniqueName: \"kubernetes.io/empty-dir/3667817b-d861-432a-8271-41f616b03cdc-bundle\") pod \"3667817b-d861-432a-8271-41f616b03cdc\" (UID: \"3667817b-d861-432a-8271-41f616b03cdc\") " Nov 01 20:01:36.507741 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:36.507630 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3667817b-d861-432a-8271-41f616b03cdc-util\" (UniqueName: \"kubernetes.io/empty-dir/3667817b-d861-432a-8271-41f616b03cdc-util\") pod \"3667817b-d861-432a-8271-41f616b03cdc\" (UID: \"3667817b-d861-432a-8271-41f616b03cdc\") " Nov 01 20:01:36.507741 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:36.507672 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3667817b-d861-432a-8271-41f616b03cdc-kube-api-access-pnhdg\" (UniqueName: \"kubernetes.io/projected/3667817b-d861-432a-8271-41f616b03cdc-kube-api-access-pnhdg\") pod \"3667817b-d861-432a-8271-41f616b03cdc\" (UID: \"3667817b-d861-432a-8271-41f616b03cdc\") " Nov 01 20:01:36.507741 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:36.507710 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58945644-f546-451e-bd8f-a3236df037f9-bundle\" (UniqueName: \"kubernetes.io/empty-dir/58945644-f546-451e-bd8f-a3236df037f9-bundle\") pod \"58945644-f546-451e-bd8f-a3236df037f9\" (UID: \"58945644-f546-451e-bd8f-a3236df037f9\") " Nov 01 20:01:36.507918 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:36.507744 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58945644-f546-451e-bd8f-a3236df037f9-kube-api-access-cww6r\" (UniqueName: \"kubernetes.io/projected/58945644-f546-451e-bd8f-a3236df037f9-kube-api-access-cww6r\") pod \"58945644-f546-451e-bd8f-a3236df037f9\" (UID: \"58945644-f546-451e-bd8f-a3236df037f9\") " Nov 01 20:01:36.507918 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:36.507786 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58945644-f546-451e-bd8f-a3236df037f9-util\" (UniqueName: \"kubernetes.io/empty-dir/58945644-f546-451e-bd8f-a3236df037f9-util\") pod \"58945644-f546-451e-bd8f-a3236df037f9\" (UID: \"58945644-f546-451e-bd8f-a3236df037f9\") " Nov 01 20:01:36.508024 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:36.507943 5935 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/aafb7b29-b974-46fa-9871-c9fbc188bb5f-util\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:01:36.508024 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:36.507961 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvkpb\" (UniqueName: \"kubernetes.io/projected/aafb7b29-b974-46fa-9871-c9fbc188bb5f-kube-api-access-lvkpb\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:01:36.508024 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:36.507959 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3667817b-d861-432a-8271-41f616b03cdc-bundle" pod "3667817b-d861-432a-8271-41f616b03cdc" (UID: "3667817b-d861-432a-8271-41f616b03cdc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:01:36.508024 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:36.507977 5935 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/aafb7b29-b974-46fa-9871-c9fbc188bb5f-bundle\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:01:36.508566 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:36.508534 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58945644-f546-451e-bd8f-a3236df037f9-bundle" pod "58945644-f546-451e-bd8f-a3236df037f9" (UID: "58945644-f546-451e-bd8f-a3236df037f9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:01:36.511008 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:36.510982 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58945644-f546-451e-bd8f-a3236df037f9-kube-api-access-cww6r" pod "58945644-f546-451e-bd8f-a3236df037f9" (UID: "58945644-f546-451e-bd8f-a3236df037f9"). InnerVolumeSpecName "kube-api-access-cww6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:01:36.511135 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:36.511012 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3667817b-d861-432a-8271-41f616b03cdc-kube-api-access-pnhdg" pod "3667817b-d861-432a-8271-41f616b03cdc" (UID: "3667817b-d861-432a-8271-41f616b03cdc"). InnerVolumeSpecName "kube-api-access-pnhdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:01:36.512919 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:36.512890 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3667817b-d861-432a-8271-41f616b03cdc-util" pod "3667817b-d861-432a-8271-41f616b03cdc" (UID: "3667817b-d861-432a-8271-41f616b03cdc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:01:36.516412 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:36.516384 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58945644-f546-451e-bd8f-a3236df037f9-util" pod "58945644-f546-451e-bd8f-a3236df037f9" (UID: "58945644-f546-451e-bd8f-a3236df037f9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:01:36.609567 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:36.609496 5935 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/58945644-f546-451e-bd8f-a3236df037f9-util\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:01:36.609567 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:36.609531 5935 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3667817b-d861-432a-8271-41f616b03cdc-bundle\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:01:36.609567 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:36.609547 5935 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3667817b-d861-432a-8271-41f616b03cdc-util\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:01:36.609567 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:36.609564 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnhdg\" (UniqueName: \"kubernetes.io/projected/3667817b-d861-432a-8271-41f616b03cdc-kube-api-access-pnhdg\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:01:36.609825 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:36.609579 5935 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/58945644-f546-451e-bd8f-a3236df037f9-bundle\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:01:36.609825 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:36.609593 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cww6r\" (UniqueName: \"kubernetes.io/projected/58945644-f546-451e-bd8f-a3236df037f9-kube-api-access-cww6r\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:01:37.107958 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:37.107920 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lnfpz" event={"ID":"aafb7b29-b974-46fa-9871-c9fbc188bb5f","Type":"ContainerDied","Data":"8fa60b2016464ba4eb67220f1e6f2a7f41fe418cb993044399b2f9c038856034"} Nov 01 20:01:37.109929 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:37.107981 5935 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8fa60b2016464ba4eb67220f1e6f2a7f41fe418cb993044399b2f9c038856034" Nov 01 20:01:37.109929 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:37.108064 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6lnfpz" Nov 01 20:01:37.117199 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:37.117173 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmkv4z" event={"ID":"1a40191e-dab8-4372-939f-ee7da2f9daa1","Type":"ContainerStarted","Data":"4e206e0df470a4b3c23ba95eaeade1002047aeb13ba5ee4894231bbc7a4331a2"} Nov 01 20:01:37.119876 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:37.119860 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7gdcm" Nov 01 20:01:37.120039 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:37.119855 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c7gdcm" event={"ID":"58945644-f546-451e-bd8f-a3236df037f9","Type":"ContainerDied","Data":"8856a2115b1dfa6a85ff2afdaa5c790b0fc6bed1bc457829b9311b100725c42f"} Nov 01 20:01:37.120129 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:37.120091 5935 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8856a2115b1dfa6a85ff2afdaa5c790b0fc6bed1bc457829b9311b100725c42f" Nov 01 20:01:37.127817 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:37.127785 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gtmn2" Nov 01 20:01:37.127926 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:37.127852 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644gtmn2" event={"ID":"3667817b-d861-432a-8271-41f616b03cdc","Type":"ContainerDied","Data":"3c8b13b45bed3141e0a9682eb1c2aaf469e6311ac83111ff15109000ffcf675c"} Nov 01 20:01:37.127926 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:37.127891 5935 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c8b13b45bed3141e0a9682eb1c2aaf469e6311ac83111ff15109000ffcf675c" Nov 01 20:01:37.149753 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:37.149700 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmkv4z" podStartSLOduration=2.86778897 podStartE2EDuration="34.149680463s" totalImagesPullingTime="31.281891493s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:01:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:01:37.148420983 +0000 UTC m=+142.921581499" watchObservedRunningTime="2026-11-01 20:01:37.149680463 +0000 UTC m=+142.922840978" Nov 01 20:01:37.282416 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:37.282389 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ff4lq" Nov 01 20:01:37.320573 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:37.320546 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atdgc9" Nov 01 20:01:37.330049 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:37.330026 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78v7xd" Nov 01 20:01:37.418745 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:37.418708 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2700c53f-9630-4fba-b2c3-ee481ecd5062-bundle\" (UniqueName: \"kubernetes.io/empty-dir/2700c53f-9630-4fba-b2c3-ee481ecd5062-bundle\") pod \"2700c53f-9630-4fba-b2c3-ee481ecd5062\" (UID: \"2700c53f-9630-4fba-b2c3-ee481ecd5062\") " Nov 01 20:01:37.418913 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:37.418785 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf4f598f-731b-4c35-85b7-86f0e700e546-bundle\" (UniqueName: \"kubernetes.io/empty-dir/cf4f598f-731b-4c35-85b7-86f0e700e546-bundle\") pod \"cf4f598f-731b-4c35-85b7-86f0e700e546\" (UID: \"cf4f598f-731b-4c35-85b7-86f0e700e546\") " Nov 01 20:01:37.418913 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:37.418836 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2700c53f-9630-4fba-b2c3-ee481ecd5062-util\" (UniqueName: \"kubernetes.io/empty-dir/2700c53f-9630-4fba-b2c3-ee481ecd5062-util\") pod \"2700c53f-9630-4fba-b2c3-ee481ecd5062\" (UID: \"2700c53f-9630-4fba-b2c3-ee481ecd5062\") " Nov 01 20:01:37.418913 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:37.418871 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf4f598f-731b-4c35-85b7-86f0e700e546-kube-api-access-plmj2\" (UniqueName: \"kubernetes.io/projected/cf4f598f-731b-4c35-85b7-86f0e700e546-kube-api-access-plmj2\") pod \"cf4f598f-731b-4c35-85b7-86f0e700e546\" (UID: \"cf4f598f-731b-4c35-85b7-86f0e700e546\") " Nov 01 20:01:37.418913 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:37.418908 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42551e2c-2573-4983-8860-f87e21da11c5-bundle\" (UniqueName: \"kubernetes.io/empty-dir/42551e2c-2573-4983-8860-f87e21da11c5-bundle\") pod \"42551e2c-2573-4983-8860-f87e21da11c5\" (UID: \"42551e2c-2573-4983-8860-f87e21da11c5\") " Nov 01 20:01:37.419133 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:37.418945 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42551e2c-2573-4983-8860-f87e21da11c5-util\" (UniqueName: \"kubernetes.io/empty-dir/42551e2c-2573-4983-8860-f87e21da11c5-util\") pod \"42551e2c-2573-4983-8860-f87e21da11c5\" (UID: \"42551e2c-2573-4983-8860-f87e21da11c5\") " Nov 01 20:01:37.419133 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:37.419007 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2700c53f-9630-4fba-b2c3-ee481ecd5062-kube-api-access-br8bm\" (UniqueName: \"kubernetes.io/projected/2700c53f-9630-4fba-b2c3-ee481ecd5062-kube-api-access-br8bm\") pod \"2700c53f-9630-4fba-b2c3-ee481ecd5062\" (UID: \"2700c53f-9630-4fba-b2c3-ee481ecd5062\") " Nov 01 20:01:37.419133 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:37.419075 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42551e2c-2573-4983-8860-f87e21da11c5-kube-api-access-5qnzm\" (UniqueName: \"kubernetes.io/projected/42551e2c-2573-4983-8860-f87e21da11c5-kube-api-access-5qnzm\") pod \"42551e2c-2573-4983-8860-f87e21da11c5\" (UID: \"42551e2c-2573-4983-8860-f87e21da11c5\") " Nov 01 20:01:37.419133 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:37.419118 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf4f598f-731b-4c35-85b7-86f0e700e546-util\" (UniqueName: \"kubernetes.io/empty-dir/cf4f598f-731b-4c35-85b7-86f0e700e546-util\") pod \"cf4f598f-731b-4c35-85b7-86f0e700e546\" (UID: \"cf4f598f-731b-4c35-85b7-86f0e700e546\") " Nov 01 20:01:37.419475 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:37.419445 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf4f598f-731b-4c35-85b7-86f0e700e546-bundle" pod "cf4f598f-731b-4c35-85b7-86f0e700e546" (UID: "cf4f598f-731b-4c35-85b7-86f0e700e546"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:01:37.419917 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:37.419869 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42551e2c-2573-4983-8860-f87e21da11c5-bundle" pod "42551e2c-2573-4983-8860-f87e21da11c5" (UID: "42551e2c-2573-4983-8860-f87e21da11c5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:01:37.420994 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:37.420964 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2700c53f-9630-4fba-b2c3-ee481ecd5062-bundle" pod "2700c53f-9630-4fba-b2c3-ee481ecd5062" (UID: "2700c53f-9630-4fba-b2c3-ee481ecd5062"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:01:37.421833 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:37.421807 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42551e2c-2573-4983-8860-f87e21da11c5-kube-api-access-5qnzm" pod "42551e2c-2573-4983-8860-f87e21da11c5" (UID: "42551e2c-2573-4983-8860-f87e21da11c5"). InnerVolumeSpecName "kube-api-access-5qnzm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:01:37.425752 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:37.425723 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf4f598f-731b-4c35-85b7-86f0e700e546-kube-api-access-plmj2" pod "cf4f598f-731b-4c35-85b7-86f0e700e546" (UID: "cf4f598f-731b-4c35-85b7-86f0e700e546"). InnerVolumeSpecName "kube-api-access-plmj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:01:37.426069 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:37.426048 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf4f598f-731b-4c35-85b7-86f0e700e546-util" pod "cf4f598f-731b-4c35-85b7-86f0e700e546" (UID: "cf4f598f-731b-4c35-85b7-86f0e700e546"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:01:37.427048 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:37.427021 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2700c53f-9630-4fba-b2c3-ee481ecd5062-kube-api-access-br8bm" pod "2700c53f-9630-4fba-b2c3-ee481ecd5062" (UID: "2700c53f-9630-4fba-b2c3-ee481ecd5062"). InnerVolumeSpecName "kube-api-access-br8bm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:01:37.428026 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:37.428001 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2700c53f-9630-4fba-b2c3-ee481ecd5062-util" pod "2700c53f-9630-4fba-b2c3-ee481ecd5062" (UID: "2700c53f-9630-4fba-b2c3-ee481ecd5062"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:01:37.428631 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:37.428604 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42551e2c-2573-4983-8860-f87e21da11c5-util" pod "42551e2c-2573-4983-8860-f87e21da11c5" (UID: "42551e2c-2573-4983-8860-f87e21da11c5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:01:37.519939 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:37.519907 5935 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2700c53f-9630-4fba-b2c3-ee481ecd5062-bundle\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:01:37.519939 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:37.519934 5935 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cf4f598f-731b-4c35-85b7-86f0e700e546-bundle\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:01:37.519939 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:37.519942 5935 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2700c53f-9630-4fba-b2c3-ee481ecd5062-util\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:01:37.520198 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:37.519951 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plmj2\" (UniqueName: \"kubernetes.io/projected/cf4f598f-731b-4c35-85b7-86f0e700e546-kube-api-access-plmj2\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:01:37.520198 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:37.519960 5935 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/42551e2c-2573-4983-8860-f87e21da11c5-bundle\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:01:37.520198 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:37.519968 5935 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/42551e2c-2573-4983-8860-f87e21da11c5-util\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:01:37.520198 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:37.519976 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-br8bm\" (UniqueName: \"kubernetes.io/projected/2700c53f-9630-4fba-b2c3-ee481ecd5062-kube-api-access-br8bm\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:01:37.520198 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:37.519984 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qnzm\" (UniqueName: \"kubernetes.io/projected/42551e2c-2573-4983-8860-f87e21da11c5-kube-api-access-5qnzm\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:01:37.520198 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:37.519992 5935 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cf4f598f-731b-4c35-85b7-86f0e700e546-util\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:01:38.134110 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:38.134078 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atdgc9" Nov 01 20:01:38.134598 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:38.134080 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847atdgc9" event={"ID":"2700c53f-9630-4fba-b2c3-ee481ecd5062","Type":"ContainerDied","Data":"65ee1d1e570101064f22181dff42df831e81015b43ef115756382fbbe281e0c6"} Nov 01 20:01:38.134598 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:38.134239 5935 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65ee1d1e570101064f22181dff42df831e81015b43ef115756382fbbe281e0c6" Nov 01 20:01:38.136061 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:38.136041 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ff4lq" Nov 01 20:01:38.136204 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:38.136032 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7ff4lq" event={"ID":"42551e2c-2573-4983-8860-f87e21da11c5","Type":"ContainerDied","Data":"a8d0f6c259d619edcc2fc083cda48e7aa7b0b6778555c27b24cb6d067563ceae"} Nov 01 20:01:38.136204 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:38.136171 5935 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8d0f6c259d619edcc2fc083cda48e7aa7b0b6778555c27b24cb6d067563ceae" Nov 01 20:01:38.137962 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:38.137939 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78v7xd" event={"ID":"cf4f598f-731b-4c35-85b7-86f0e700e546","Type":"ContainerDied","Data":"dcc9cef2f33773331bb924d7e1cfb16d884f47654f148ebe4c01a7f61eeae2a3"} Nov 01 20:01:38.138073 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:38.137978 5935 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dcc9cef2f33773331bb924d7e1cfb16d884f47654f148ebe4c01a7f61eeae2a3" Nov 01 20:01:38.138073 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:38.137949 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d78v7xd" Nov 01 20:01:38.140199 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:38.140177 5935 generic.go:356] "Generic (PLEG): container finished" podID="1a40191e-dab8-4372-939f-ee7da2f9daa1" containerID="4e206e0df470a4b3c23ba95eaeade1002047aeb13ba5ee4894231bbc7a4331a2" exitCode=0 Nov 01 20:01:38.140320 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:38.140213 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmkv4z" event={"ID":"1a40191e-dab8-4372-939f-ee7da2f9daa1","Type":"ContainerDied","Data":"4e206e0df470a4b3c23ba95eaeade1002047aeb13ba5ee4894231bbc7a4331a2"} Nov 01 20:01:39.263410 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:39.263388 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmkv4z" Nov 01 20:01:39.438437 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:39.438352 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a40191e-dab8-4372-939f-ee7da2f9daa1-util\" (UniqueName: \"kubernetes.io/empty-dir/1a40191e-dab8-4372-939f-ee7da2f9daa1-util\") pod \"1a40191e-dab8-4372-939f-ee7da2f9daa1\" (UID: \"1a40191e-dab8-4372-939f-ee7da2f9daa1\") " Nov 01 20:01:39.438591 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:39.438441 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a40191e-dab8-4372-939f-ee7da2f9daa1-kube-api-access-mssmx\" (UniqueName: \"kubernetes.io/projected/1a40191e-dab8-4372-939f-ee7da2f9daa1-kube-api-access-mssmx\") pod \"1a40191e-dab8-4372-939f-ee7da2f9daa1\" (UID: \"1a40191e-dab8-4372-939f-ee7da2f9daa1\") " Nov 01 20:01:39.438591 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:39.438468 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a40191e-dab8-4372-939f-ee7da2f9daa1-bundle\" (UniqueName: \"kubernetes.io/empty-dir/1a40191e-dab8-4372-939f-ee7da2f9daa1-bundle\") pod \"1a40191e-dab8-4372-939f-ee7da2f9daa1\" (UID: \"1a40191e-dab8-4372-939f-ee7da2f9daa1\") " Nov 01 20:01:39.439094 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:39.439062 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a40191e-dab8-4372-939f-ee7da2f9daa1-bundle" pod "1a40191e-dab8-4372-939f-ee7da2f9daa1" (UID: "1a40191e-dab8-4372-939f-ee7da2f9daa1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:01:39.440425 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:39.440398 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a40191e-dab8-4372-939f-ee7da2f9daa1-kube-api-access-mssmx" pod "1a40191e-dab8-4372-939f-ee7da2f9daa1" (UID: "1a40191e-dab8-4372-939f-ee7da2f9daa1"). InnerVolumeSpecName "kube-api-access-mssmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:01:39.444399 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:39.444300 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a40191e-dab8-4372-939f-ee7da2f9daa1-util" pod "1a40191e-dab8-4372-939f-ee7da2f9daa1" (UID: "1a40191e-dab8-4372-939f-ee7da2f9daa1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:01:39.539127 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:39.539102 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mssmx\" (UniqueName: \"kubernetes.io/projected/1a40191e-dab8-4372-939f-ee7da2f9daa1-kube-api-access-mssmx\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:01:39.539127 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:39.539127 5935 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1a40191e-dab8-4372-939f-ee7da2f9daa1-bundle\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:01:39.539261 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:39.539138 5935 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1a40191e-dab8-4372-939f-ee7da2f9daa1-util\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:01:40.153777 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:40.153735 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmkv4z" event={"ID":"1a40191e-dab8-4372-939f-ee7da2f9daa1","Type":"ContainerDied","Data":"41128a07d4c1681f35b5d28dc628fe1fbca235250f6392bc0420fcd6ead508cd"} Nov 01 20:01:40.153961 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:40.153792 5935 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41128a07d4c1681f35b5d28dc628fe1fbca235250f6392bc0420fcd6ead508cd" Nov 01 20:01:40.153961 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:40.153739 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cmkv4z" Nov 01 20:01:40.155456 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:40.155426 5935 generic.go:356] "Generic (PLEG): container finished" podID="48a12663-97c4-486c-9462-183947bbfdc1" containerID="c31b4b4f1bfe491355b2759b6fd05f724e9b6c64ae37ccd6bb93cf4a324621f6" exitCode=0 Nov 01 20:01:40.155586 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:40.155506 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-lvm4r" event={"ID":"48a12663-97c4-486c-9462-183947bbfdc1","Type":"ContainerDied","Data":"c31b4b4f1bfe491355b2759b6fd05f724e9b6c64ae37ccd6bb93cf4a324621f6"} Nov 01 20:01:41.162210 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:41.162174 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-lvm4r" event={"ID":"48a12663-97c4-486c-9462-183947bbfdc1","Type":"ContainerStarted","Data":"505f4bcd1db2d9cf8286ab171eb5e40f651591ace43c509e356cb95ba498b159"} Nov 01 20:01:41.162646 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:41.162329 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-lvm4r" Nov 01 20:01:41.204281 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:41.204229 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-lvm4r" podStartSLOduration=2.543060826 podStartE2EDuration="7.204212514s" totalImagesPullingTime="4.189375233s" totalInitContainerRuntime="471.776455ms" isStatefulPod=false podCreationTimestamp="2026-11-01 20:01:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:01:41.202209435 +0000 UTC m=+146.975369947" watchObservedRunningTime="2026-11-01 20:01:41.204212514 +0000 UTC m=+146.977373414" Nov 01 20:01:52.172164 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:52.172135 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-689bcf9f9-lvm4r" Nov 01 20:01:52.281694 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:52.281663 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-qjf26"] Nov 01 20:01:52.281968 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:52.281934 5935 cpu_manager.go:414] "containerMap: removing container" podUID="3667817b-d861-432a-8271-41f616b03cdc" containerName="util" Nov 01 20:01:52.281968 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:52.281947 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3667817b-d861-432a-8271-41f616b03cdc" containerName="util" Nov 01 20:01:52.282092 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:52.281977 5935 cpu_manager.go:414] "containerMap: removing container" podUID="1a40191e-dab8-4372-939f-ee7da2f9daa1" containerName="extract" Nov 01 20:01:52.282092 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:52.281983 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a40191e-dab8-4372-939f-ee7da2f9daa1" containerName="extract" Nov 01 20:01:52.282092 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:52.281997 5935 cpu_manager.go:414] "containerMap: removing container" podUID="cf4f598f-731b-4c35-85b7-86f0e700e546" containerName="util" Nov 01 20:01:52.282092 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:52.282002 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf4f598f-731b-4c35-85b7-86f0e700e546" containerName="util" Nov 01 20:01:52.282092 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:52.282008 5935 cpu_manager.go:414] "containerMap: removing container" podUID="42551e2c-2573-4983-8860-f87e21da11c5" containerName="util" Nov 01 20:01:52.282092 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:52.282012 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42551e2c-2573-4983-8860-f87e21da11c5" containerName="util" Nov 01 20:01:52.282092 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:52.282020 5935 cpu_manager.go:414] "containerMap: removing container" podUID="58945644-f546-451e-bd8f-a3236df037f9" containerName="pull" Nov 01 20:01:52.282092 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:52.282025 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58945644-f546-451e-bd8f-a3236df037f9" containerName="pull" Nov 01 20:01:52.282092 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:52.282031 5935 cpu_manager.go:414] "containerMap: removing container" podUID="3667817b-d861-432a-8271-41f616b03cdc" containerName="pull" Nov 01 20:01:52.282092 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:52.282035 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3667817b-d861-432a-8271-41f616b03cdc" containerName="pull" Nov 01 20:01:52.282092 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:52.282041 5935 cpu_manager.go:414] "containerMap: removing container" podUID="aafb7b29-b974-46fa-9871-c9fbc188bb5f" containerName="pull" Nov 01 20:01:52.282092 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:52.282045 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aafb7b29-b974-46fa-9871-c9fbc188bb5f" containerName="pull" Nov 01 20:01:52.282092 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:52.282051 5935 cpu_manager.go:414] "containerMap: removing container" podUID="1a40191e-dab8-4372-939f-ee7da2f9daa1" containerName="pull" Nov 01 20:01:52.282092 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:52.282055 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a40191e-dab8-4372-939f-ee7da2f9daa1" containerName="pull" Nov 01 20:01:52.282092 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:52.282063 5935 cpu_manager.go:414] "containerMap: removing container" podUID="cf4f598f-731b-4c35-85b7-86f0e700e546" containerName="extract" Nov 01 20:01:52.282092 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:52.282067 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf4f598f-731b-4c35-85b7-86f0e700e546" containerName="extract" Nov 01 20:01:52.282092 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:52.282076 5935 cpu_manager.go:414] "containerMap: removing container" podUID="aafb7b29-b974-46fa-9871-c9fbc188bb5f" containerName="util" Nov 01 20:01:52.282092 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:52.282080 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aafb7b29-b974-46fa-9871-c9fbc188bb5f" containerName="util" Nov 01 20:01:52.282092 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:52.282085 5935 cpu_manager.go:414] "containerMap: removing container" podUID="42551e2c-2573-4983-8860-f87e21da11c5" containerName="extract" Nov 01 20:01:52.282092 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:52.282089 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42551e2c-2573-4983-8860-f87e21da11c5" containerName="extract" Nov 01 20:01:52.282092 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:52.282098 5935 cpu_manager.go:414] "containerMap: removing container" podUID="cf4f598f-731b-4c35-85b7-86f0e700e546" containerName="pull" Nov 01 20:01:52.282718 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:52.282102 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf4f598f-731b-4c35-85b7-86f0e700e546" containerName="pull" Nov 01 20:01:52.282718 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:52.282109 5935 cpu_manager.go:414] "containerMap: removing container" podUID="2700c53f-9630-4fba-b2c3-ee481ecd5062" containerName="util" Nov 01 20:01:52.282718 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:52.282114 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2700c53f-9630-4fba-b2c3-ee481ecd5062" containerName="util" Nov 01 20:01:52.282718 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:52.282120 5935 cpu_manager.go:414] "containerMap: removing container" podUID="42551e2c-2573-4983-8860-f87e21da11c5" containerName="pull" Nov 01 20:01:52.282718 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:52.282123 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42551e2c-2573-4983-8860-f87e21da11c5" containerName="pull" Nov 01 20:01:52.282718 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:52.282131 5935 cpu_manager.go:414] "containerMap: removing container" podUID="58945644-f546-451e-bd8f-a3236df037f9" containerName="extract" Nov 01 20:01:52.282718 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:52.282135 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58945644-f546-451e-bd8f-a3236df037f9" containerName="extract" Nov 01 20:01:52.282718 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:52.282142 5935 cpu_manager.go:414] "containerMap: removing container" podUID="2700c53f-9630-4fba-b2c3-ee481ecd5062" containerName="pull" Nov 01 20:01:52.282718 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:52.282146 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2700c53f-9630-4fba-b2c3-ee481ecd5062" containerName="pull" Nov 01 20:01:52.282718 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:52.282155 5935 cpu_manager.go:414] "containerMap: removing container" podUID="3667817b-d861-432a-8271-41f616b03cdc" containerName="extract" Nov 01 20:01:52.282718 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:52.282159 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3667817b-d861-432a-8271-41f616b03cdc" containerName="extract" Nov 01 20:01:52.282718 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:52.282167 5935 cpu_manager.go:414] "containerMap: removing container" podUID="58945644-f546-451e-bd8f-a3236df037f9" containerName="util" Nov 01 20:01:52.282718 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:52.282171 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58945644-f546-451e-bd8f-a3236df037f9" containerName="util" Nov 01 20:01:52.282718 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:52.282177 5935 cpu_manager.go:414] "containerMap: removing container" podUID="aafb7b29-b974-46fa-9871-c9fbc188bb5f" containerName="extract" Nov 01 20:01:52.282718 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:52.282181 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aafb7b29-b974-46fa-9871-c9fbc188bb5f" containerName="extract" Nov 01 20:01:52.282718 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:52.282187 5935 cpu_manager.go:414] "containerMap: removing container" podUID="2700c53f-9630-4fba-b2c3-ee481ecd5062" containerName="extract" Nov 01 20:01:52.282718 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:52.282191 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2700c53f-9630-4fba-b2c3-ee481ecd5062" containerName="extract" Nov 01 20:01:52.282718 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:52.282197 5935 cpu_manager.go:414] "containerMap: removing container" podUID="1a40191e-dab8-4372-939f-ee7da2f9daa1" containerName="util" Nov 01 20:01:52.282718 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:52.282201 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a40191e-dab8-4372-939f-ee7da2f9daa1" containerName="util" Nov 01 20:01:52.282718 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:52.282246 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="3667817b-d861-432a-8271-41f616b03cdc" containerName="extract" Nov 01 20:01:52.282718 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:52.282254 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="aafb7b29-b974-46fa-9871-c9fbc188bb5f" containerName="extract" Nov 01 20:01:52.282718 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:52.282259 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="2700c53f-9630-4fba-b2c3-ee481ecd5062" containerName="extract" Nov 01 20:01:52.282718 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:52.282265 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a40191e-dab8-4372-939f-ee7da2f9daa1" containerName="extract" Nov 01 20:01:52.282718 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:52.282272 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf4f598f-731b-4c35-85b7-86f0e700e546" containerName="extract" Nov 01 20:01:52.282718 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:52.282278 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="42551e2c-2573-4983-8860-f87e21da11c5" containerName="extract" Nov 01 20:01:52.282718 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:52.282303 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="58945644-f546-451e-bd8f-a3236df037f9" containerName="extract" Nov 01 20:01:52.287909 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:52.287892 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-qjf26" Nov 01 20:01:52.295590 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:52.295564 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Nov 01 20:01:52.296439 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:52.296419 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Nov 01 20:01:52.296549 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:52.296503 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-2lcnj\"" Nov 01 20:01:52.297203 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:52.297187 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-qjf26"] Nov 01 20:01:52.339163 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:52.339141 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79ng8\" (UniqueName: \"kubernetes.io/projected/4bb94c2d-5860-429b-8797-69769787f57d-kube-api-access-79ng8\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-qjf26\" (UID: \"4bb94c2d-5860-429b-8797-69769787f57d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-qjf26" Nov 01 20:01:52.339268 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:52.339176 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4bb94c2d-5860-429b-8797-69769787f57d-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-qjf26\" (UID: \"4bb94c2d-5860-429b-8797-69769787f57d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-qjf26" Nov 01 20:01:52.440386 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:52.440303 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79ng8\" (UniqueName: \"kubernetes.io/projected/4bb94c2d-5860-429b-8797-69769787f57d-kube-api-access-79ng8\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-qjf26\" (UID: \"4bb94c2d-5860-429b-8797-69769787f57d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-qjf26" Nov 01 20:01:52.440386 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:52.440358 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4bb94c2d-5860-429b-8797-69769787f57d-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-qjf26\" (UID: \"4bb94c2d-5860-429b-8797-69769787f57d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-qjf26" Nov 01 20:01:52.440698 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:52.440681 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4bb94c2d-5860-429b-8797-69769787f57d-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-qjf26\" (UID: \"4bb94c2d-5860-429b-8797-69769787f57d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-qjf26" Nov 01 20:01:52.447805 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:52.447780 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79ng8\" (UniqueName: \"kubernetes.io/projected/4bb94c2d-5860-429b-8797-69769787f57d-kube-api-access-79ng8\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-qjf26\" (UID: \"4bb94c2d-5860-429b-8797-69769787f57d\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-qjf26" Nov 01 20:01:52.600330 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:52.600300 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-qjf26" Nov 01 20:01:52.873363 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:52.873336 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-qjf26"] Nov 01 20:01:52.876366 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:01:52.876335 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bb94c2d_5860_429b_8797_69769787f57d.slice/crio-aa890867c12bb58e6ef760bf253ea3a4d0e887ac6169dc6d930071c7bc9de902 WatchSource:0}: Error finding container aa890867c12bb58e6ef760bf253ea3a4d0e887ac6169dc6d930071c7bc9de902: Status 404 returned error can't find the container with id aa890867c12bb58e6ef760bf253ea3a4d0e887ac6169dc6d930071c7bc9de902 Nov 01 20:01:53.216878 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:53.216848 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-qjf26" event={"ID":"4bb94c2d-5860-429b-8797-69769787f57d","Type":"ContainerStarted","Data":"aa890867c12bb58e6ef760bf253ea3a4d0e887ac6169dc6d930071c7bc9de902"} Nov 01 20:01:53.576800 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:53.576719 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-sxxmt"] Nov 01 20:01:53.581404 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:53.581384 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-sxxmt" Nov 01 20:01:53.584828 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:53.584811 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-swjbx\"" Nov 01 20:01:53.585884 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:53.585861 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 01 20:01:53.585992 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:53.585900 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 01 20:01:53.589731 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:53.589711 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-sxxmt"] Nov 01 20:01:53.649978 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:53.649952 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b0ff9d9c-30ed-4a38-9558-63e72aefcf39-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-sxxmt\" (UID: \"b0ff9d9c-30ed-4a38-9558-63e72aefcf39\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-sxxmt" Nov 01 20:01:53.650092 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:53.650015 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9dls\" (UniqueName: \"kubernetes.io/projected/b0ff9d9c-30ed-4a38-9558-63e72aefcf39-kube-api-access-r9dls\") pod \"kuadrant-operator-controller-manager-cf8469d7b-sxxmt\" (UID: \"b0ff9d9c-30ed-4a38-9558-63e72aefcf39\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-sxxmt" Nov 01 20:01:53.750358 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:53.750329 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9dls\" (UniqueName: \"kubernetes.io/projected/b0ff9d9c-30ed-4a38-9558-63e72aefcf39-kube-api-access-r9dls\") pod \"kuadrant-operator-controller-manager-cf8469d7b-sxxmt\" (UID: \"b0ff9d9c-30ed-4a38-9558-63e72aefcf39\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-sxxmt" Nov 01 20:01:53.750459 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:53.750379 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b0ff9d9c-30ed-4a38-9558-63e72aefcf39-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-sxxmt\" (UID: \"b0ff9d9c-30ed-4a38-9558-63e72aefcf39\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-sxxmt" Nov 01 20:01:53.750789 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:53.750772 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b0ff9d9c-30ed-4a38-9558-63e72aefcf39-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-sxxmt\" (UID: \"b0ff9d9c-30ed-4a38-9558-63e72aefcf39\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-sxxmt" Nov 01 20:01:53.761940 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:53.761912 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9dls\" (UniqueName: \"kubernetes.io/projected/b0ff9d9c-30ed-4a38-9558-63e72aefcf39-kube-api-access-r9dls\") pod \"kuadrant-operator-controller-manager-cf8469d7b-sxxmt\" (UID: \"b0ff9d9c-30ed-4a38-9558-63e72aefcf39\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-sxxmt" Nov 01 20:01:53.894807 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:53.894733 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-sxxmt" Nov 01 20:01:54.174231 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:54.174202 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-sxxmt"] Nov 01 20:01:54.177221 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:01:54.177192 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0ff9d9c_30ed_4a38_9558_63e72aefcf39.slice/crio-212bac4dbc6a5538688c11336c62760d8994941ac614575367c51e5ff7fd3131 WatchSource:0}: Error finding container 212bac4dbc6a5538688c11336c62760d8994941ac614575367c51e5ff7fd3131: Status 404 returned error can't find the container with id 212bac4dbc6a5538688c11336c62760d8994941ac614575367c51e5ff7fd3131 Nov 01 20:01:54.221712 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:54.221683 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-sxxmt" event={"ID":"b0ff9d9c-30ed-4a38-9558-63e72aefcf39","Type":"ContainerStarted","Data":"212bac4dbc6a5538688c11336c62760d8994941ac614575367c51e5ff7fd3131"} Nov 01 20:01:56.232342 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:56.232309 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-qjf26" event={"ID":"4bb94c2d-5860-429b-8797-69769787f57d","Type":"ContainerStarted","Data":"f017d86f28e765ecf158d2d4af391af74ea3e73afcb18a90c7bde9fe428cc961"} Nov 01 20:01:56.254989 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:56.254936 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-qjf26" podStartSLOduration=1.897738857 podStartE2EDuration="4.254920721s" totalImagesPullingTime="2.357181864s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:01:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:01:56.253102363 +0000 UTC m=+162.026262886" watchObservedRunningTime="2026-11-01 20:01:56.254920721 +0000 UTC m=+162.028081233" Nov 01 20:01:56.728260 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:56.728224 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-mcmdt"] Nov 01 20:01:56.731943 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:56.731918 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-mcmdt" Nov 01 20:01:56.741090 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:56.741051 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-8tnls\"" Nov 01 20:01:56.743280 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:56.743243 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Nov 01 20:01:56.743413 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:56.743306 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Nov 01 20:01:56.744805 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:56.744783 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-mcmdt"] Nov 01 20:01:56.776832 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:56.776809 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqlkq\" (UniqueName: \"kubernetes.io/projected/c68e7c85-46f6-408d-8296-bcdbd72ba817-kube-api-access-tqlkq\") pod \"jobset-operator-674cd59dc5-mcmdt\" (UID: \"c68e7c85-46f6-408d-8296-bcdbd72ba817\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-mcmdt" Nov 01 20:01:56.776965 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:56.776842 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c68e7c85-46f6-408d-8296-bcdbd72ba817-tmp\") pod \"jobset-operator-674cd59dc5-mcmdt\" (UID: \"c68e7c85-46f6-408d-8296-bcdbd72ba817\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-mcmdt" Nov 01 20:01:56.877602 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:56.877569 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqlkq\" (UniqueName: \"kubernetes.io/projected/c68e7c85-46f6-408d-8296-bcdbd72ba817-kube-api-access-tqlkq\") pod \"jobset-operator-674cd59dc5-mcmdt\" (UID: \"c68e7c85-46f6-408d-8296-bcdbd72ba817\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-mcmdt" Nov 01 20:01:56.877602 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:56.877604 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c68e7c85-46f6-408d-8296-bcdbd72ba817-tmp\") pod \"jobset-operator-674cd59dc5-mcmdt\" (UID: \"c68e7c85-46f6-408d-8296-bcdbd72ba817\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-mcmdt" Nov 01 20:01:56.877975 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:56.877957 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c68e7c85-46f6-408d-8296-bcdbd72ba817-tmp\") pod \"jobset-operator-674cd59dc5-mcmdt\" (UID: \"c68e7c85-46f6-408d-8296-bcdbd72ba817\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-mcmdt" Nov 01 20:01:56.892440 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:56.892410 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqlkq\" (UniqueName: \"kubernetes.io/projected/c68e7c85-46f6-408d-8296-bcdbd72ba817-kube-api-access-tqlkq\") pod \"jobset-operator-674cd59dc5-mcmdt\" (UID: \"c68e7c85-46f6-408d-8296-bcdbd72ba817\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-mcmdt" Nov 01 20:01:57.057571 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:57.057492 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-mcmdt" Nov 01 20:01:57.344782 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:57.344744 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-mcmdt"] Nov 01 20:01:57.347481 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:01:57.347449 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc68e7c85_46f6_408d_8296_bcdbd72ba817.slice/crio-d4d942e22dd5ec8d582966025ed581c6b202af5ebfe06d3d21d877c8dd1b0567 WatchSource:0}: Error finding container d4d942e22dd5ec8d582966025ed581c6b202af5ebfe06d3d21d877c8dd1b0567: Status 404 returned error can't find the container with id d4d942e22dd5ec8d582966025ed581c6b202af5ebfe06d3d21d877c8dd1b0567 Nov 01 20:01:58.245572 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:58.245531 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-mcmdt" event={"ID":"c68e7c85-46f6-408d-8296-bcdbd72ba817","Type":"ContainerStarted","Data":"d4d942e22dd5ec8d582966025ed581c6b202af5ebfe06d3d21d877c8dd1b0567"} Nov 01 20:01:59.808591 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:59.808558 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-fxgzj"] Nov 01 20:01:59.811999 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:59.811978 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-fxgzj" Nov 01 20:01:59.822223 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:59.822203 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-cbndd\"" Nov 01 20:01:59.822348 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:59.822236 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 01 20:01:59.823307 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:59.823277 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 01 20:01:59.824584 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:59.824567 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-fxgzj"] Nov 01 20:01:59.904465 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:59.904419 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eda4922a-91c6-42bc-82ee-f0da1a36888f-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-fxgzj\" (UID: \"eda4922a-91c6-42bc-82ee-f0da1a36888f\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-fxgzj" Nov 01 20:01:59.904638 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:01:59.904470 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l96kz\" (UniqueName: \"kubernetes.io/projected/eda4922a-91c6-42bc-82ee-f0da1a36888f-kube-api-access-l96kz\") pod \"cert-manager-cainjector-5c57bbd4fc-fxgzj\" (UID: \"eda4922a-91c6-42bc-82ee-f0da1a36888f\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-fxgzj" Nov 01 20:02:00.005393 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:00.005357 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eda4922a-91c6-42bc-82ee-f0da1a36888f-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-fxgzj\" (UID: \"eda4922a-91c6-42bc-82ee-f0da1a36888f\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-fxgzj" Nov 01 20:02:00.005393 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:00.005396 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l96kz\" (UniqueName: \"kubernetes.io/projected/eda4922a-91c6-42bc-82ee-f0da1a36888f-kube-api-access-l96kz\") pod \"cert-manager-cainjector-5c57bbd4fc-fxgzj\" (UID: \"eda4922a-91c6-42bc-82ee-f0da1a36888f\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-fxgzj" Nov 01 20:02:00.014672 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:00.014647 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l96kz\" (UniqueName: \"kubernetes.io/projected/eda4922a-91c6-42bc-82ee-f0da1a36888f-kube-api-access-l96kz\") pod \"cert-manager-cainjector-5c57bbd4fc-fxgzj\" (UID: \"eda4922a-91c6-42bc-82ee-f0da1a36888f\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-fxgzj" Nov 01 20:02:00.018007 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:00.017978 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eda4922a-91c6-42bc-82ee-f0da1a36888f-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-fxgzj\" (UID: \"eda4922a-91c6-42bc-82ee-f0da1a36888f\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-fxgzj" Nov 01 20:02:00.123305 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:00.123225 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-fxgzj" Nov 01 20:02:00.412386 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:00.412086 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-fxgzj"] Nov 01 20:02:00.414666 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:02:00.414638 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeda4922a_91c6_42bc_82ee_f0da1a36888f.slice/crio-d0f2261823db555733636bc14cd9c969324074c18fefe0bf55e967d2899f3068 WatchSource:0}: Error finding container d0f2261823db555733636bc14cd9c969324074c18fefe0bf55e967d2899f3068: Status 404 returned error can't find the container with id d0f2261823db555733636bc14cd9c969324074c18fefe0bf55e967d2899f3068 Nov 01 20:02:01.260816 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:01.260777 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-mcmdt" event={"ID":"c68e7c85-46f6-408d-8296-bcdbd72ba817","Type":"ContainerStarted","Data":"2514c222895365ed46838e32f5cd01340b498b0b064e866c90e415b0930afdf9"} Nov 01 20:02:01.262114 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:01.262087 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-fxgzj" event={"ID":"eda4922a-91c6-42bc-82ee-f0da1a36888f","Type":"ContainerStarted","Data":"d0f2261823db555733636bc14cd9c969324074c18fefe0bf55e967d2899f3068"} Nov 01 20:02:01.286752 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:01.286705 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-mcmdt" podStartSLOduration=1.717979366 podStartE2EDuration="5.286691841s" totalImagesPullingTime="3.568712475s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:01:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:02:01.286486728 +0000 UTC m=+167.059647238" watchObservedRunningTime="2026-11-01 20:02:01.286691841 +0000 UTC m=+167.059852353" Nov 01 20:02:02.762011 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:02.760524 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86fbgwkn"] Nov 01 20:02:02.764932 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:02.764867 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86fbgwkn" Nov 01 20:02:02.774044 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:02.774012 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-nsq8m\"" Nov 01 20:02:02.789615 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:02.789568 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86fbgwkn"] Nov 01 20:02:02.834571 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:02.834538 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87d067a6-b0e8-46d7-92f8-df20c468a672-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-79f98cb86fbgwkn\" (UID: \"87d067a6-b0e8-46d7-92f8-df20c468a672\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86fbgwkn" Nov 01 20:02:02.834760 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:02.834601 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87d067a6-b0e8-46d7-92f8-df20c468a672-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-79f98cb86fbgwkn\" (UID: \"87d067a6-b0e8-46d7-92f8-df20c468a672\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86fbgwkn" Nov 01 20:02:02.834760 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:02.834702 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87d067a6-b0e8-46d7-92f8-df20c468a672-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-79f98cb86fbgwkn\" (UID: \"87d067a6-b0e8-46d7-92f8-df20c468a672\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86fbgwkn" Nov 01 20:02:02.834760 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:02.834746 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87d067a6-b0e8-46d7-92f8-df20c468a672-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-79f98cb86fbgwkn\" (UID: \"87d067a6-b0e8-46d7-92f8-df20c468a672\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86fbgwkn" Nov 01 20:02:02.834889 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:02.834765 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87d067a6-b0e8-46d7-92f8-df20c468a672-istio-data\") pod \"data-science-gateway-data-science-gateway-class-79f98cb86fbgwkn\" (UID: \"87d067a6-b0e8-46d7-92f8-df20c468a672\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86fbgwkn" Nov 01 20:02:02.834889 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:02.834786 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87d067a6-b0e8-46d7-92f8-df20c468a672-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-79f98cb86fbgwkn\" (UID: \"87d067a6-b0e8-46d7-92f8-df20c468a672\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86fbgwkn" Nov 01 20:02:02.834889 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:02.834830 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87d067a6-b0e8-46d7-92f8-df20c468a672-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-79f98cb86fbgwkn\" (UID: \"87d067a6-b0e8-46d7-92f8-df20c468a672\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86fbgwkn" Nov 01 20:02:02.834889 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:02.834854 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcqmj\" (UniqueName: \"kubernetes.io/projected/87d067a6-b0e8-46d7-92f8-df20c468a672-kube-api-access-bcqmj\") pod \"data-science-gateway-data-science-gateway-class-79f98cb86fbgwkn\" (UID: \"87d067a6-b0e8-46d7-92f8-df20c468a672\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86fbgwkn" Nov 01 20:02:02.835051 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:02.834900 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87d067a6-b0e8-46d7-92f8-df20c468a672-istio-token\") pod \"data-science-gateway-data-science-gateway-class-79f98cb86fbgwkn\" (UID: \"87d067a6-b0e8-46d7-92f8-df20c468a672\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86fbgwkn" Nov 01 20:02:02.935909 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:02.935875 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87d067a6-b0e8-46d7-92f8-df20c468a672-istio-token\") pod \"data-science-gateway-data-science-gateway-class-79f98cb86fbgwkn\" (UID: \"87d067a6-b0e8-46d7-92f8-df20c468a672\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86fbgwkn" Nov 01 20:02:02.936088 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:02.935925 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87d067a6-b0e8-46d7-92f8-df20c468a672-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-79f98cb86fbgwkn\" (UID: \"87d067a6-b0e8-46d7-92f8-df20c468a672\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86fbgwkn" Nov 01 20:02:02.936088 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:02.935973 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87d067a6-b0e8-46d7-92f8-df20c468a672-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-79f98cb86fbgwkn\" (UID: \"87d067a6-b0e8-46d7-92f8-df20c468a672\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86fbgwkn" Nov 01 20:02:02.936088 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:02.936003 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87d067a6-b0e8-46d7-92f8-df20c468a672-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-79f98cb86fbgwkn\" (UID: \"87d067a6-b0e8-46d7-92f8-df20c468a672\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86fbgwkn" Nov 01 20:02:02.936088 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:02.936036 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87d067a6-b0e8-46d7-92f8-df20c468a672-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-79f98cb86fbgwkn\" (UID: \"87d067a6-b0e8-46d7-92f8-df20c468a672\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86fbgwkn" Nov 01 20:02:02.936088 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:02.936060 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87d067a6-b0e8-46d7-92f8-df20c468a672-istio-data\") pod \"data-science-gateway-data-science-gateway-class-79f98cb86fbgwkn\" (UID: \"87d067a6-b0e8-46d7-92f8-df20c468a672\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86fbgwkn" Nov 01 20:02:02.936088 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:02.936083 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87d067a6-b0e8-46d7-92f8-df20c468a672-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-79f98cb86fbgwkn\" (UID: \"87d067a6-b0e8-46d7-92f8-df20c468a672\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86fbgwkn" Nov 01 20:02:02.936448 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:02.936141 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87d067a6-b0e8-46d7-92f8-df20c468a672-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-79f98cb86fbgwkn\" (UID: \"87d067a6-b0e8-46d7-92f8-df20c468a672\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86fbgwkn" Nov 01 20:02:02.936448 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:02.936164 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcqmj\" (UniqueName: \"kubernetes.io/projected/87d067a6-b0e8-46d7-92f8-df20c468a672-kube-api-access-bcqmj\") pod \"data-science-gateway-data-science-gateway-class-79f98cb86fbgwkn\" (UID: \"87d067a6-b0e8-46d7-92f8-df20c468a672\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86fbgwkn" Nov 01 20:02:02.936712 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:02.936687 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87d067a6-b0e8-46d7-92f8-df20c468a672-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-79f98cb86fbgwkn\" (UID: \"87d067a6-b0e8-46d7-92f8-df20c468a672\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86fbgwkn" Nov 01 20:02:02.936781 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:02.936743 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87d067a6-b0e8-46d7-92f8-df20c468a672-istio-data\") pod \"data-science-gateway-data-science-gateway-class-79f98cb86fbgwkn\" (UID: \"87d067a6-b0e8-46d7-92f8-df20c468a672\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86fbgwkn" Nov 01 20:02:02.937025 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:02.937005 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87d067a6-b0e8-46d7-92f8-df20c468a672-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-79f98cb86fbgwkn\" (UID: \"87d067a6-b0e8-46d7-92f8-df20c468a672\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86fbgwkn" Nov 01 20:02:02.937215 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:02.937191 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87d067a6-b0e8-46d7-92f8-df20c468a672-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-79f98cb86fbgwkn\" (UID: \"87d067a6-b0e8-46d7-92f8-df20c468a672\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86fbgwkn" Nov 01 20:02:02.937332 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:02.937209 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87d067a6-b0e8-46d7-92f8-df20c468a672-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-79f98cb86fbgwkn\" (UID: \"87d067a6-b0e8-46d7-92f8-df20c468a672\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86fbgwkn" Nov 01 20:02:02.939577 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:02.939549 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87d067a6-b0e8-46d7-92f8-df20c468a672-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-79f98cb86fbgwkn\" (UID: \"87d067a6-b0e8-46d7-92f8-df20c468a672\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86fbgwkn" Nov 01 20:02:02.950080 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:02.950032 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87d067a6-b0e8-46d7-92f8-df20c468a672-istio-token\") pod \"data-science-gateway-data-science-gateway-class-79f98cb86fbgwkn\" (UID: \"87d067a6-b0e8-46d7-92f8-df20c468a672\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86fbgwkn" Nov 01 20:02:02.950330 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:02.950301 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87d067a6-b0e8-46d7-92f8-df20c468a672-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-79f98cb86fbgwkn\" (UID: \"87d067a6-b0e8-46d7-92f8-df20c468a672\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86fbgwkn" Nov 01 20:02:02.950651 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:02.950623 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcqmj\" (UniqueName: \"kubernetes.io/projected/87d067a6-b0e8-46d7-92f8-df20c468a672-kube-api-access-bcqmj\") pod \"data-science-gateway-data-science-gateway-class-79f98cb86fbgwkn\" (UID: \"87d067a6-b0e8-46d7-92f8-df20c468a672\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86fbgwkn" Nov 01 20:02:03.085436 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:03.085339 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86fbgwkn" Nov 01 20:02:03.446476 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:03.446430 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86fbgwkn"] Nov 01 20:02:03.452373 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:02:03.452339 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87d067a6_b0e8_46d7_92f8_df20c468a672.slice/crio-d2d3986c84347794450916b53f93a389ed01909aed47927fbcebe36557080d74 WatchSource:0}: Error finding container d2d3986c84347794450916b53f93a389ed01909aed47927fbcebe36557080d74: Status 404 returned error can't find the container with id d2d3986c84347794450916b53f93a389ed01909aed47927fbcebe36557080d74 Nov 01 20:02:03.455928 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:03.455419 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:02:03.455928 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:03.455511 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:02:03.455928 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:03.455555 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:02:03.879422 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:03.879338 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-vjdvl"] Nov 01 20:02:03.894606 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:03.894574 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-vjdvl"] Nov 01 20:02:03.894775 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:03.894673 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vjdvl" Nov 01 20:02:03.943570 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:03.943535 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rxmb\" (UniqueName: \"kubernetes.io/projected/07d598f4-98bd-43bf-b5b8-2bdfd4b1431a-kube-api-access-2rxmb\") pod \"cert-manager-webhook-5dbbbf75cc-vjdvl\" (UID: \"07d598f4-98bd-43bf-b5b8-2bdfd4b1431a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vjdvl" Nov 01 20:02:03.943718 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:03.943622 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/07d598f4-98bd-43bf-b5b8-2bdfd4b1431a-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-vjdvl\" (UID: \"07d598f4-98bd-43bf-b5b8-2bdfd4b1431a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vjdvl" Nov 01 20:02:03.945266 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:03.945241 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-fq5x9\"" Nov 01 20:02:04.045057 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:04.045021 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rxmb\" (UniqueName: \"kubernetes.io/projected/07d598f4-98bd-43bf-b5b8-2bdfd4b1431a-kube-api-access-2rxmb\") pod \"cert-manager-webhook-5dbbbf75cc-vjdvl\" (UID: \"07d598f4-98bd-43bf-b5b8-2bdfd4b1431a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vjdvl" Nov 01 20:02:04.045237 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:04.045077 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/07d598f4-98bd-43bf-b5b8-2bdfd4b1431a-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-vjdvl\" (UID: \"07d598f4-98bd-43bf-b5b8-2bdfd4b1431a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vjdvl" Nov 01 20:02:04.058701 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:04.058679 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/07d598f4-98bd-43bf-b5b8-2bdfd4b1431a-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-vjdvl\" (UID: \"07d598f4-98bd-43bf-b5b8-2bdfd4b1431a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vjdvl" Nov 01 20:02:04.058858 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:04.058839 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rxmb\" (UniqueName: \"kubernetes.io/projected/07d598f4-98bd-43bf-b5b8-2bdfd4b1431a-kube-api-access-2rxmb\") pod \"cert-manager-webhook-5dbbbf75cc-vjdvl\" (UID: \"07d598f4-98bd-43bf-b5b8-2bdfd4b1431a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vjdvl" Nov 01 20:02:04.223751 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:04.223718 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vjdvl" Nov 01 20:02:04.282370 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:04.282334 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86fbgwkn" event={"ID":"87d067a6-b0e8-46d7-92f8-df20c468a672","Type":"ContainerStarted","Data":"1bbfcbbaf6d840d8528931c50a94d68aac4cb36c73beee94ff83b34897a8b473"} Nov 01 20:02:04.282530 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:04.282397 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86fbgwkn" event={"ID":"87d067a6-b0e8-46d7-92f8-df20c468a672","Type":"ContainerStarted","Data":"d2d3986c84347794450916b53f93a389ed01909aed47927fbcebe36557080d74"} Nov 01 20:02:04.308503 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:04.308451 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86fbgwkn" podStartSLOduration=2.308432798 podStartE2EDuration="2.308432798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:02:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:02:04.307193653 +0000 UTC m=+170.080354206" watchObservedRunningTime="2026-11-01 20:02:04.308432798 +0000 UTC m=+170.081593311" Nov 01 20:02:04.562610 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:04.562492 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-vjdvl"] Nov 01 20:02:04.714683 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:02:04.714650 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod07d598f4_98bd_43bf_b5b8_2bdfd4b1431a.slice/crio-641fbb7e194d25367dc8336ed074ad805dfade06d0ea6c2f1cda2940e467706a WatchSource:0}: Error finding container 641fbb7e194d25367dc8336ed074ad805dfade06d0ea6c2f1cda2940e467706a: Status 404 returned error can't find the container with id 641fbb7e194d25367dc8336ed074ad805dfade06d0ea6c2f1cda2940e467706a Nov 01 20:02:05.086442 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:05.086362 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86fbgwkn" Nov 01 20:02:05.091594 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:05.091571 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86fbgwkn" Nov 01 20:02:05.288773 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:05.288736 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-fxgzj" event={"ID":"eda4922a-91c6-42bc-82ee-f0da1a36888f","Type":"ContainerStarted","Data":"890f0f481bc48d3c6a785b5b8428f76019bd91481aad4a9b9bd4a237708d1a7f"} Nov 01 20:02:05.291220 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:05.291177 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vjdvl" event={"ID":"07d598f4-98bd-43bf-b5b8-2bdfd4b1431a","Type":"ContainerStarted","Data":"641fbb7e194d25367dc8336ed074ad805dfade06d0ea6c2f1cda2940e467706a"} Nov 01 20:02:05.295677 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:05.295528 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-sxxmt" event={"ID":"b0ff9d9c-30ed-4a38-9558-63e72aefcf39","Type":"ContainerStarted","Data":"574b19879bf2372cf40df3ae5d26c1c91eb8f0d0e4a926c353a7602c3bce278c"} Nov 01 20:02:05.295677 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:05.295626 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86fbgwkn" Nov 01 20:02:05.295845 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:05.295792 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-sxxmt" Nov 01 20:02:05.297762 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:05.296411 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-79f98cb86fbgwkn" Nov 01 20:02:05.307409 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:05.307363 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-fxgzj" podStartSLOduration=1.974006266 podStartE2EDuration="6.307351798s" totalImagesPullingTime="4.333345532s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:01:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:02:05.307073969 +0000 UTC m=+171.080234482" watchObservedRunningTime="2026-11-01 20:02:05.307351798 +0000 UTC m=+171.080512310" Nov 01 20:02:05.352538 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:05.352428 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-sxxmt" podStartSLOduration=2.236783259 podStartE2EDuration="12.352409224s" totalImagesPullingTime="10.115625965s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:01:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:02:05.350099812 +0000 UTC m=+171.123260325" watchObservedRunningTime="2026-11-01 20:02:05.352409224 +0000 UTC m=+171.125569737" Nov 01 20:02:05.843597 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:02:05.843481 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:02:06.300507 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:06.300463 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vjdvl" event={"ID":"07d598f4-98bd-43bf-b5b8-2bdfd4b1431a","Type":"ContainerStarted","Data":"0c4856a08578d14dc39d851b9528a5543a3986c2e66fe097f41b3a6870eb647b"} Nov 01 20:02:07.304808 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:07.304781 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vjdvl" Nov 01 20:02:07.765400 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:07.765346 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vjdvl" podStartSLOduration=3.866830253 podStartE2EDuration="4.765331205s" totalImagesPullingTime="898.500952ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:02:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:02:06.322992304 +0000 UTC m=+172.096152828" watchObservedRunningTime="2026-11-01 20:02:07.765331205 +0000 UTC m=+173.538491721" Nov 01 20:02:07.765627 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:07.765613 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-rfv55"] Nov 01 20:02:07.767938 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:07.767919 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-rfv55" Nov 01 20:02:07.777059 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:07.777035 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-fn4xx\"" Nov 01 20:02:07.778882 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:07.778854 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-rfv55"] Nov 01 20:02:07.878080 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:07.878050 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpsgt\" (UniqueName: \"kubernetes.io/projected/0f1e6aa9-ece8-4636-858a-4061042eb8bd-kube-api-access-xpsgt\") pod \"cert-manager-597b95cb84-rfv55\" (UID: \"0f1e6aa9-ece8-4636-858a-4061042eb8bd\") " pod="cert-manager/cert-manager-597b95cb84-rfv55" Nov 01 20:02:07.878238 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:07.878119 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0f1e6aa9-ece8-4636-858a-4061042eb8bd-bound-sa-token\") pod \"cert-manager-597b95cb84-rfv55\" (UID: \"0f1e6aa9-ece8-4636-858a-4061042eb8bd\") " pod="cert-manager/cert-manager-597b95cb84-rfv55" Nov 01 20:02:07.979139 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:07.979105 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0f1e6aa9-ece8-4636-858a-4061042eb8bd-bound-sa-token\") pod \"cert-manager-597b95cb84-rfv55\" (UID: \"0f1e6aa9-ece8-4636-858a-4061042eb8bd\") " pod="cert-manager/cert-manager-597b95cb84-rfv55" Nov 01 20:02:07.979344 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:07.979164 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpsgt\" (UniqueName: \"kubernetes.io/projected/0f1e6aa9-ece8-4636-858a-4061042eb8bd-kube-api-access-xpsgt\") pod \"cert-manager-597b95cb84-rfv55\" (UID: \"0f1e6aa9-ece8-4636-858a-4061042eb8bd\") " pod="cert-manager/cert-manager-597b95cb84-rfv55" Nov 01 20:02:07.987678 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:07.987653 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0f1e6aa9-ece8-4636-858a-4061042eb8bd-bound-sa-token\") pod \"cert-manager-597b95cb84-rfv55\" (UID: \"0f1e6aa9-ece8-4636-858a-4061042eb8bd\") " pod="cert-manager/cert-manager-597b95cb84-rfv55" Nov 01 20:02:07.987793 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:07.987773 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpsgt\" (UniqueName: \"kubernetes.io/projected/0f1e6aa9-ece8-4636-858a-4061042eb8bd-kube-api-access-xpsgt\") pod \"cert-manager-597b95cb84-rfv55\" (UID: \"0f1e6aa9-ece8-4636-858a-4061042eb8bd\") " pod="cert-manager/cert-manager-597b95cb84-rfv55" Nov 01 20:02:08.079910 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:08.079816 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-rfv55" Nov 01 20:02:08.350974 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:02:08.350946 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f1e6aa9_ece8_4636_858a_4061042eb8bd.slice/crio-192896351b59e66411a45cee945c6ef0a262dc7497bae6e82b7c36c0d56c2db0 WatchSource:0}: Error finding container 192896351b59e66411a45cee945c6ef0a262dc7497bae6e82b7c36c0d56c2db0: Status 404 returned error can't find the container with id 192896351b59e66411a45cee945c6ef0a262dc7497bae6e82b7c36c0d56c2db0 Nov 01 20:02:08.352542 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:08.352518 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-rfv55"] Nov 01 20:02:09.315340 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:09.315304 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-rfv55" event={"ID":"0f1e6aa9-ece8-4636-858a-4061042eb8bd","Type":"ContainerStarted","Data":"0c4696ed24351885a1faafa61c60f381b33e4256052a3ed3ebd24ae95ecdfdbf"} Nov 01 20:02:09.315520 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:09.315346 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-rfv55" event={"ID":"0f1e6aa9-ece8-4636-858a-4061042eb8bd","Type":"ContainerStarted","Data":"192896351b59e66411a45cee945c6ef0a262dc7497bae6e82b7c36c0d56c2db0"} Nov 01 20:02:09.330506 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:09.330460 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-rfv55" podStartSLOduration=2.330446801 podStartE2EDuration="2.330446801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:02:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:02:09.329834809 +0000 UTC m=+175.102995322" watchObservedRunningTime="2026-11-01 20:02:09.330446801 +0000 UTC m=+175.103607312" Nov 01 20:02:13.311481 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:13.311449 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-vjdvl" Nov 01 20:02:16.303053 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:16.303023 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-sxxmt" Nov 01 20:02:27.713404 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:27.713317 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-7bbf46bdb7-57pw7"] Nov 01 20:02:27.716716 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:27.716700 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-7bbf46bdb7-57pw7" Nov 01 20:02:27.723968 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:27.723942 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-7bbf46bdb7-57pw7"] Nov 01 20:02:27.724958 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:27.724929 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Nov 01 20:02:27.725627 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:27.725607 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Nov 01 20:02:27.725697 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:27.725658 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Nov 01 20:02:27.725697 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:27.725674 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-htpfr\"" Nov 01 20:02:27.858000 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:27.857970 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/05b8db82-93af-4b55-9454-d1e0df8a9aee-manager-config\") pod \"jobset-controller-manager-7bbf46bdb7-57pw7\" (UID: \"05b8db82-93af-4b55-9454-d1e0df8a9aee\") " pod="openshift-jobset-operator/jobset-controller-manager-7bbf46bdb7-57pw7" Nov 01 20:02:27.858168 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:27.858036 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05b8db82-93af-4b55-9454-d1e0df8a9aee-metrics-certs\") pod \"jobset-controller-manager-7bbf46bdb7-57pw7\" (UID: \"05b8db82-93af-4b55-9454-d1e0df8a9aee\") " pod="openshift-jobset-operator/jobset-controller-manager-7bbf46bdb7-57pw7" Nov 01 20:02:27.858168 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:27.858146 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7lxx\" (UniqueName: \"kubernetes.io/projected/05b8db82-93af-4b55-9454-d1e0df8a9aee-kube-api-access-s7lxx\") pod \"jobset-controller-manager-7bbf46bdb7-57pw7\" (UID: \"05b8db82-93af-4b55-9454-d1e0df8a9aee\") " pod="openshift-jobset-operator/jobset-controller-manager-7bbf46bdb7-57pw7" Nov 01 20:02:27.858241 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:27.858176 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/05b8db82-93af-4b55-9454-d1e0df8a9aee-cert\") pod \"jobset-controller-manager-7bbf46bdb7-57pw7\" (UID: \"05b8db82-93af-4b55-9454-d1e0df8a9aee\") " pod="openshift-jobset-operator/jobset-controller-manager-7bbf46bdb7-57pw7" Nov 01 20:02:27.959385 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:27.959357 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7lxx\" (UniqueName: \"kubernetes.io/projected/05b8db82-93af-4b55-9454-d1e0df8a9aee-kube-api-access-s7lxx\") pod \"jobset-controller-manager-7bbf46bdb7-57pw7\" (UID: \"05b8db82-93af-4b55-9454-d1e0df8a9aee\") " pod="openshift-jobset-operator/jobset-controller-manager-7bbf46bdb7-57pw7" Nov 01 20:02:27.959537 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:27.959397 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/05b8db82-93af-4b55-9454-d1e0df8a9aee-cert\") pod \"jobset-controller-manager-7bbf46bdb7-57pw7\" (UID: \"05b8db82-93af-4b55-9454-d1e0df8a9aee\") " pod="openshift-jobset-operator/jobset-controller-manager-7bbf46bdb7-57pw7" Nov 01 20:02:27.959537 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:27.959428 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/05b8db82-93af-4b55-9454-d1e0df8a9aee-manager-config\") pod \"jobset-controller-manager-7bbf46bdb7-57pw7\" (UID: \"05b8db82-93af-4b55-9454-d1e0df8a9aee\") " pod="openshift-jobset-operator/jobset-controller-manager-7bbf46bdb7-57pw7" Nov 01 20:02:27.959629 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:27.959552 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05b8db82-93af-4b55-9454-d1e0df8a9aee-metrics-certs\") pod \"jobset-controller-manager-7bbf46bdb7-57pw7\" (UID: \"05b8db82-93af-4b55-9454-d1e0df8a9aee\") " pod="openshift-jobset-operator/jobset-controller-manager-7bbf46bdb7-57pw7" Nov 01 20:02:27.960054 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:27.960029 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/05b8db82-93af-4b55-9454-d1e0df8a9aee-manager-config\") pod \"jobset-controller-manager-7bbf46bdb7-57pw7\" (UID: \"05b8db82-93af-4b55-9454-d1e0df8a9aee\") " pod="openshift-jobset-operator/jobset-controller-manager-7bbf46bdb7-57pw7" Nov 01 20:02:27.961720 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:27.961695 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/05b8db82-93af-4b55-9454-d1e0df8a9aee-cert\") pod \"jobset-controller-manager-7bbf46bdb7-57pw7\" (UID: \"05b8db82-93af-4b55-9454-d1e0df8a9aee\") " pod="openshift-jobset-operator/jobset-controller-manager-7bbf46bdb7-57pw7" Nov 01 20:02:27.961842 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:27.961824 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05b8db82-93af-4b55-9454-d1e0df8a9aee-metrics-certs\") pod \"jobset-controller-manager-7bbf46bdb7-57pw7\" (UID: \"05b8db82-93af-4b55-9454-d1e0df8a9aee\") " pod="openshift-jobset-operator/jobset-controller-manager-7bbf46bdb7-57pw7" Nov 01 20:02:27.967015 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:27.966970 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7lxx\" (UniqueName: \"kubernetes.io/projected/05b8db82-93af-4b55-9454-d1e0df8a9aee-kube-api-access-s7lxx\") pod \"jobset-controller-manager-7bbf46bdb7-57pw7\" (UID: \"05b8db82-93af-4b55-9454-d1e0df8a9aee\") " pod="openshift-jobset-operator/jobset-controller-manager-7bbf46bdb7-57pw7" Nov 01 20:02:28.029023 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:28.029001 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-7bbf46bdb7-57pw7" Nov 01 20:02:28.309577 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:28.309502 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-7bbf46bdb7-57pw7"] Nov 01 20:02:28.312403 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:02:28.312374 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05b8db82_93af_4b55_9454_d1e0df8a9aee.slice/crio-e0c7cf353611169e9af163634aab6ea757ae91614118f7a3a19fdd32287ab73a WatchSource:0}: Error finding container e0c7cf353611169e9af163634aab6ea757ae91614118f7a3a19fdd32287ab73a: Status 404 returned error can't find the container with id e0c7cf353611169e9af163634aab6ea757ae91614118f7a3a19fdd32287ab73a Nov 01 20:02:28.399211 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:28.399178 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-7bbf46bdb7-57pw7" event={"ID":"05b8db82-93af-4b55-9454-d1e0df8a9aee","Type":"ContainerStarted","Data":"e0c7cf353611169e9af163634aab6ea757ae91614118f7a3a19fdd32287ab73a"} Nov 01 20:02:33.430228 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:33.430191 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-7bbf46bdb7-57pw7" event={"ID":"05b8db82-93af-4b55-9454-d1e0df8a9aee","Type":"ContainerStarted","Data":"9f9d737410f3226c65f056fa445e7144b7b2378a22d0a80b2d71d5120675ebc6"} Nov 01 20:02:33.430679 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:33.430343 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-7bbf46bdb7-57pw7" Nov 01 20:02:33.447098 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:33.447054 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-7bbf46bdb7-57pw7" podStartSLOduration=1.492952587 podStartE2EDuration="6.447042124s" totalImagesPullingTime="4.954089537s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:02:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:02:33.446368697 +0000 UTC m=+199.219529209" watchObservedRunningTime="2026-11-01 20:02:33.447042124 +0000 UTC m=+199.220202636" Nov 01 20:02:33.511509 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:33.511468 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-sxxmt"] Nov 01 20:02:33.511740 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:33.511713 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-sxxmt" podUID="b0ff9d9c-30ed-4a38-9558-63e72aefcf39" containerName="manager" containerID="cri-o://574b19879bf2372cf40df3ae5d26c1c91eb8f0d0e4a926c353a7602c3bce278c" gracePeriod=10 Nov 01 20:02:33.766053 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:33.766027 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-sxxmt" Nov 01 20:02:33.912602 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:33.912532 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0ff9d9c-30ed-4a38-9558-63e72aefcf39-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b0ff9d9c-30ed-4a38-9558-63e72aefcf39-extensions-socket-volume\") pod \"b0ff9d9c-30ed-4a38-9558-63e72aefcf39\" (UID: \"b0ff9d9c-30ed-4a38-9558-63e72aefcf39\") " Nov 01 20:02:33.912602 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:33.912595 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0ff9d9c-30ed-4a38-9558-63e72aefcf39-kube-api-access-r9dls\" (UniqueName: \"kubernetes.io/projected/b0ff9d9c-30ed-4a38-9558-63e72aefcf39-kube-api-access-r9dls\") pod \"b0ff9d9c-30ed-4a38-9558-63e72aefcf39\" (UID: \"b0ff9d9c-30ed-4a38-9558-63e72aefcf39\") " Nov 01 20:02:33.912948 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:33.912923 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0ff9d9c-30ed-4a38-9558-63e72aefcf39-extensions-socket-volume" pod "b0ff9d9c-30ed-4a38-9558-63e72aefcf39" (UID: "b0ff9d9c-30ed-4a38-9558-63e72aefcf39"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:02:33.914786 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:33.914756 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0ff9d9c-30ed-4a38-9558-63e72aefcf39-kube-api-access-r9dls" pod "b0ff9d9c-30ed-4a38-9558-63e72aefcf39" (UID: "b0ff9d9c-30ed-4a38-9558-63e72aefcf39"). InnerVolumeSpecName "kube-api-access-r9dls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:02:34.013424 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:34.013400 5935 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b0ff9d9c-30ed-4a38-9558-63e72aefcf39-extensions-socket-volume\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:02:34.013424 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:34.013422 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9dls\" (UniqueName: \"kubernetes.io/projected/b0ff9d9c-30ed-4a38-9558-63e72aefcf39-kube-api-access-r9dls\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:02:34.436079 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:34.436041 5935 generic.go:356] "Generic (PLEG): container finished" podID="b0ff9d9c-30ed-4a38-9558-63e72aefcf39" containerID="574b19879bf2372cf40df3ae5d26c1c91eb8f0d0e4a926c353a7602c3bce278c" exitCode=0 Nov 01 20:02:34.436486 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:34.436106 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-sxxmt" Nov 01 20:02:34.436486 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:34.436105 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-sxxmt" event={"ID":"b0ff9d9c-30ed-4a38-9558-63e72aefcf39","Type":"ContainerDied","Data":"574b19879bf2372cf40df3ae5d26c1c91eb8f0d0e4a926c353a7602c3bce278c"} Nov 01 20:02:34.436486 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:34.436163 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-sxxmt" event={"ID":"b0ff9d9c-30ed-4a38-9558-63e72aefcf39","Type":"ContainerDied","Data":"212bac4dbc6a5538688c11336c62760d8994941ac614575367c51e5ff7fd3131"} Nov 01 20:02:34.436486 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:34.436189 5935 scope.go:122] "RemoveContainer" containerID="574b19879bf2372cf40df3ae5d26c1c91eb8f0d0e4a926c353a7602c3bce278c" Nov 01 20:02:34.444905 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:34.444884 5935 scope.go:122] "RemoveContainer" containerID="574b19879bf2372cf40df3ae5d26c1c91eb8f0d0e4a926c353a7602c3bce278c" Nov 01 20:02:34.445203 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:02:34.445173 5935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"574b19879bf2372cf40df3ae5d26c1c91eb8f0d0e4a926c353a7602c3bce278c\": container with ID starting with 574b19879bf2372cf40df3ae5d26c1c91eb8f0d0e4a926c353a7602c3bce278c not found: ID does not exist" containerID="574b19879bf2372cf40df3ae5d26c1c91eb8f0d0e4a926c353a7602c3bce278c" Nov 01 20:02:34.445311 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:34.445213 5935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"574b19879bf2372cf40df3ae5d26c1c91eb8f0d0e4a926c353a7602c3bce278c"} err="failed to get container status \"574b19879bf2372cf40df3ae5d26c1c91eb8f0d0e4a926c353a7602c3bce278c\": rpc error: code = NotFound desc = could not find container \"574b19879bf2372cf40df3ae5d26c1c91eb8f0d0e4a926c353a7602c3bce278c\": container with ID starting with 574b19879bf2372cf40df3ae5d26c1c91eb8f0d0e4a926c353a7602c3bce278c not found: ID does not exist" Nov 01 20:02:34.458404 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:34.458383 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-sxxmt"] Nov 01 20:02:34.470434 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:34.470406 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-sxxmt"] Nov 01 20:02:35.842949 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:02:35.842904 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:02:36.351596 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:36.351566 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0ff9d9c-30ed-4a38-9558-63e72aefcf39" path="/var/lib/kubelet/pods/b0ff9d9c-30ed-4a38-9558-63e72aefcf39/volumes" Nov 01 20:02:44.441942 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:44.441914 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-7bbf46bdb7-57pw7" Nov 01 20:02:51.497878 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:51.497836 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-jtc4h"] Nov 01 20:02:51.498393 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:51.498356 5935 cpu_manager.go:414] "containerMap: removing container" podUID="b0ff9d9c-30ed-4a38-9558-63e72aefcf39" containerName="manager" Nov 01 20:02:51.498393 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:51.498376 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0ff9d9c-30ed-4a38-9558-63e72aefcf39" containerName="manager" Nov 01 20:02:51.498531 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:51.498516 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0ff9d9c-30ed-4a38-9558-63e72aefcf39" containerName="manager" Nov 01 20:02:51.501249 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:51.501228 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-jtc4h" Nov 01 20:02:51.507744 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:51.507698 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-4pgmt\"" Nov 01 20:02:51.511853 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:51.511828 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-jtc4h"] Nov 01 20:02:51.512045 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:51.512024 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Nov 01 20:02:51.512143 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:51.512123 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 01 20:02:51.512205 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:51.512185 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 01 20:02:51.552821 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:51.552793 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9pb7\" (UniqueName: \"kubernetes.io/projected/cc6156d4-83e7-4da4-aa28-3b026753a86c-kube-api-access-j9pb7\") pod \"limitador-limitador-74dfcfb894-jtc4h\" (UID: \"cc6156d4-83e7-4da4-aa28-3b026753a86c\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-jtc4h" Nov 01 20:02:51.552931 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:51.552832 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/cc6156d4-83e7-4da4-aa28-3b026753a86c-config-file\") pod \"limitador-limitador-74dfcfb894-jtc4h\" (UID: \"cc6156d4-83e7-4da4-aa28-3b026753a86c\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-jtc4h" Nov 01 20:02:51.592684 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:51.592657 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-jtc4h"] Nov 01 20:02:51.654307 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:51.654259 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9pb7\" (UniqueName: \"kubernetes.io/projected/cc6156d4-83e7-4da4-aa28-3b026753a86c-kube-api-access-j9pb7\") pod \"limitador-limitador-74dfcfb894-jtc4h\" (UID: \"cc6156d4-83e7-4da4-aa28-3b026753a86c\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-jtc4h" Nov 01 20:02:51.654457 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:51.654330 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/cc6156d4-83e7-4da4-aa28-3b026753a86c-config-file\") pod \"limitador-limitador-74dfcfb894-jtc4h\" (UID: \"cc6156d4-83e7-4da4-aa28-3b026753a86c\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-jtc4h" Nov 01 20:02:51.654936 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:51.654919 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/cc6156d4-83e7-4da4-aa28-3b026753a86c-config-file\") pod \"limitador-limitador-74dfcfb894-jtc4h\" (UID: \"cc6156d4-83e7-4da4-aa28-3b026753a86c\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-jtc4h" Nov 01 20:02:51.662797 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:51.662773 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9pb7\" (UniqueName: \"kubernetes.io/projected/cc6156d4-83e7-4da4-aa28-3b026753a86c-kube-api-access-j9pb7\") pod \"limitador-limitador-74dfcfb894-jtc4h\" (UID: \"cc6156d4-83e7-4da4-aa28-3b026753a86c\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-jtc4h" Nov 01 20:02:51.814570 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:51.814498 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-jtc4h" Nov 01 20:02:52.091048 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:52.091024 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-jtc4h"] Nov 01 20:02:52.093680 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:02:52.093646 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc6156d4_83e7_4da4_aa28_3b026753a86c.slice/crio-94b925fc887500b32b0993c510f72d640ca8a9f2f6bcf036272b003891c63401 WatchSource:0}: Error finding container 94b925fc887500b32b0993c510f72d640ca8a9f2f6bcf036272b003891c63401: Status 404 returned error can't find the container with id 94b925fc887500b32b0993c510f72d640ca8a9f2f6bcf036272b003891c63401 Nov 01 20:02:52.519900 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:52.519865 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-jtc4h" event={"ID":"cc6156d4-83e7-4da4-aa28-3b026753a86c","Type":"ContainerStarted","Data":"94b925fc887500b32b0993c510f72d640ca8a9f2f6bcf036272b003891c63401"} Nov 01 20:02:53.526229 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:53.526199 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-jtc4h" event={"ID":"cc6156d4-83e7-4da4-aa28-3b026753a86c","Type":"ContainerStarted","Data":"090a0dbc2d9274e417593452decfccc4198b763ec062afd72928feb85133b854"} Nov 01 20:02:53.526634 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:53.526326 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-jtc4h" Nov 01 20:02:53.555717 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:53.555677 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-jtc4h" podStartSLOduration=1.223152947 podStartE2EDuration="2.555665101s" totalImagesPullingTime="1.332512154s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:02:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:02:53.553991529 +0000 UTC m=+219.327152039" watchObservedRunningTime="2026-11-01 20:02:53.555665101 +0000 UTC m=+219.328825649" Nov 01 20:02:56.640327 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:56.640276 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-x9p74"] Nov 01 20:02:56.643822 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:56.643802 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-x9p74" Nov 01 20:02:56.652541 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:56.652522 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-k2bt6\"" Nov 01 20:02:56.652677 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:56.652554 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 01 20:02:56.652677 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:56.652523 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 01 20:02:56.652677 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:56.652587 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Nov 01 20:02:56.654141 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:56.654121 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-x9p74"] Nov 01 20:02:56.658788 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:56.658768 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-w6g7x"] Nov 01 20:02:56.662229 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:56.662212 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-w6g7x" Nov 01 20:02:56.669359 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:56.669338 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Nov 01 20:02:56.670741 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:56.670721 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-pjf8j\"" Nov 01 20:02:56.670850 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:56.670726 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-w6g7x"] Nov 01 20:02:56.682348 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:56.682323 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-jbfsn"] Nov 01 20:02:56.685528 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:56.685512 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-jbfsn" Nov 01 20:02:56.696528 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:56.696505 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg79t\" (UniqueName: \"kubernetes.io/projected/e4de1911-27fd-43cb-970b-308f8380bd6d-kube-api-access-dg79t\") pod \"kserve-controller-manager-776cc66d47-x9p74\" (UID: \"e4de1911-27fd-43cb-970b-308f8380bd6d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-x9p74" Nov 01 20:02:56.696687 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:56.696596 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e4de1911-27fd-43cb-970b-308f8380bd6d-cert\") pod \"kserve-controller-manager-776cc66d47-x9p74\" (UID: \"e4de1911-27fd-43cb-970b-308f8380bd6d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-x9p74" Nov 01 20:02:56.698473 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:56.698451 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-jbfsn"] Nov 01 20:02:56.742349 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:56.742319 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-mxlw6\"" Nov 01 20:02:56.742349 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:56.742339 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Nov 01 20:02:56.797708 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:56.797672 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/277af67c-a53b-48c6-9cd0-41c1af1c95dd-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-w6g7x\" (UID: \"277af67c-a53b-48c6-9cd0-41c1af1c95dd\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-w6g7x" Nov 01 20:02:56.797831 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:56.797728 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q94j6\" (UniqueName: \"kubernetes.io/projected/277af67c-a53b-48c6-9cd0-41c1af1c95dd-kube-api-access-q94j6\") pod \"llmisvc-controller-manager-869d5dcc5d-w6g7x\" (UID: \"277af67c-a53b-48c6-9cd0-41c1af1c95dd\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-w6g7x" Nov 01 20:02:56.797880 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:56.797824 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg79t\" (UniqueName: \"kubernetes.io/projected/e4de1911-27fd-43cb-970b-308f8380bd6d-kube-api-access-dg79t\") pod \"kserve-controller-manager-776cc66d47-x9p74\" (UID: \"e4de1911-27fd-43cb-970b-308f8380bd6d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-x9p74" Nov 01 20:02:56.797880 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:56.797873 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33b28b24-2fcc-40d9-9ad0-059f15acf4aa-tls-certs\") pod \"model-serving-api-75bd89fdfc-jbfsn\" (UID: \"33b28b24-2fcc-40d9-9ad0-059f15acf4aa\") " pod="opendatahub/model-serving-api-75bd89fdfc-jbfsn" Nov 01 20:02:56.797953 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:56.797893 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45swn\" (UniqueName: \"kubernetes.io/projected/33b28b24-2fcc-40d9-9ad0-059f15acf4aa-kube-api-access-45swn\") pod \"model-serving-api-75bd89fdfc-jbfsn\" (UID: \"33b28b24-2fcc-40d9-9ad0-059f15acf4aa\") " pod="opendatahub/model-serving-api-75bd89fdfc-jbfsn" Nov 01 20:02:56.798006 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:56.797955 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e4de1911-27fd-43cb-970b-308f8380bd6d-cert\") pod \"kserve-controller-manager-776cc66d47-x9p74\" (UID: \"e4de1911-27fd-43cb-970b-308f8380bd6d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-x9p74" Nov 01 20:02:56.800884 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:56.800853 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e4de1911-27fd-43cb-970b-308f8380bd6d-cert\") pod \"kserve-controller-manager-776cc66d47-x9p74\" (UID: \"e4de1911-27fd-43cb-970b-308f8380bd6d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-x9p74" Nov 01 20:02:56.811769 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:56.811748 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg79t\" (UniqueName: \"kubernetes.io/projected/e4de1911-27fd-43cb-970b-308f8380bd6d-kube-api-access-dg79t\") pod \"kserve-controller-manager-776cc66d47-x9p74\" (UID: \"e4de1911-27fd-43cb-970b-308f8380bd6d\") " pod="opendatahub/kserve-controller-manager-776cc66d47-x9p74" Nov 01 20:02:56.898591 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:56.898503 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/277af67c-a53b-48c6-9cd0-41c1af1c95dd-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-w6g7x\" (UID: \"277af67c-a53b-48c6-9cd0-41c1af1c95dd\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-w6g7x" Nov 01 20:02:56.898591 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:56.898560 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q94j6\" (UniqueName: \"kubernetes.io/projected/277af67c-a53b-48c6-9cd0-41c1af1c95dd-kube-api-access-q94j6\") pod \"llmisvc-controller-manager-869d5dcc5d-w6g7x\" (UID: \"277af67c-a53b-48c6-9cd0-41c1af1c95dd\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-w6g7x" Nov 01 20:02:56.898769 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:56.898604 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33b28b24-2fcc-40d9-9ad0-059f15acf4aa-tls-certs\") pod \"model-serving-api-75bd89fdfc-jbfsn\" (UID: \"33b28b24-2fcc-40d9-9ad0-059f15acf4aa\") " pod="opendatahub/model-serving-api-75bd89fdfc-jbfsn" Nov 01 20:02:56.898769 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:56.898622 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45swn\" (UniqueName: \"kubernetes.io/projected/33b28b24-2fcc-40d9-9ad0-059f15acf4aa-kube-api-access-45swn\") pod \"model-serving-api-75bd89fdfc-jbfsn\" (UID: \"33b28b24-2fcc-40d9-9ad0-059f15acf4aa\") " pod="opendatahub/model-serving-api-75bd89fdfc-jbfsn" Nov 01 20:02:56.900886 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:56.900858 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/277af67c-a53b-48c6-9cd0-41c1af1c95dd-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-w6g7x\" (UID: \"277af67c-a53b-48c6-9cd0-41c1af1c95dd\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-w6g7x" Nov 01 20:02:56.900994 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:56.900967 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33b28b24-2fcc-40d9-9ad0-059f15acf4aa-tls-certs\") pod \"model-serving-api-75bd89fdfc-jbfsn\" (UID: \"33b28b24-2fcc-40d9-9ad0-059f15acf4aa\") " pod="opendatahub/model-serving-api-75bd89fdfc-jbfsn" Nov 01 20:02:56.916219 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:56.916194 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q94j6\" (UniqueName: \"kubernetes.io/projected/277af67c-a53b-48c6-9cd0-41c1af1c95dd-kube-api-access-q94j6\") pod \"llmisvc-controller-manager-869d5dcc5d-w6g7x\" (UID: \"277af67c-a53b-48c6-9cd0-41c1af1c95dd\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-w6g7x" Nov 01 20:02:56.916353 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:56.916267 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45swn\" (UniqueName: \"kubernetes.io/projected/33b28b24-2fcc-40d9-9ad0-059f15acf4aa-kube-api-access-45swn\") pod \"model-serving-api-75bd89fdfc-jbfsn\" (UID: \"33b28b24-2fcc-40d9-9ad0-059f15acf4aa\") " pod="opendatahub/model-serving-api-75bd89fdfc-jbfsn" Nov 01 20:02:56.958042 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:56.958015 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-x9p74" Nov 01 20:02:56.976928 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:56.976905 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-w6g7x" Nov 01 20:02:56.999642 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:56.999621 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-jbfsn" Nov 01 20:02:57.235447 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:57.235424 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-x9p74"] Nov 01 20:02:57.237639 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:02:57.237607 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4de1911_27fd_43cb_970b_308f8380bd6d.slice/crio-58b9f039fee0e1eb910a2efefd2bf72a244f89d914b8ae95384c18141a7c8cd9 WatchSource:0}: Error finding container 58b9f039fee0e1eb910a2efefd2bf72a244f89d914b8ae95384c18141a7c8cd9: Status 404 returned error can't find the container with id 58b9f039fee0e1eb910a2efefd2bf72a244f89d914b8ae95384c18141a7c8cd9 Nov 01 20:02:57.260071 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:57.260052 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-w6g7x"] Nov 01 20:02:57.261980 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:02:57.261959 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod277af67c_a53b_48c6_9cd0_41c1af1c95dd.slice/crio-e5e5f01ea3b8af6170e8a648a307d53c622712ad343436d7559bc0515e5871e1 WatchSource:0}: Error finding container e5e5f01ea3b8af6170e8a648a307d53c622712ad343436d7559bc0515e5871e1: Status 404 returned error can't find the container with id e5e5f01ea3b8af6170e8a648a307d53c622712ad343436d7559bc0515e5871e1 Nov 01 20:02:57.292725 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:57.292704 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-jbfsn"] Nov 01 20:02:57.294588 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:02:57.294562 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33b28b24_2fcc_40d9_9ad0_059f15acf4aa.slice/crio-7703c7be22ef098247d6a8f2e56f7dabc373f1670382a810c529284a8951781b WatchSource:0}: Error finding container 7703c7be22ef098247d6a8f2e56f7dabc373f1670382a810c529284a8951781b: Status 404 returned error can't find the container with id 7703c7be22ef098247d6a8f2e56f7dabc373f1670382a810c529284a8951781b Nov 01 20:02:57.548280 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:57.548207 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-jbfsn" event={"ID":"33b28b24-2fcc-40d9-9ad0-059f15acf4aa","Type":"ContainerStarted","Data":"7703c7be22ef098247d6a8f2e56f7dabc373f1670382a810c529284a8951781b"} Nov 01 20:02:57.549318 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:57.549298 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-x9p74" event={"ID":"e4de1911-27fd-43cb-970b-308f8380bd6d","Type":"ContainerStarted","Data":"58b9f039fee0e1eb910a2efefd2bf72a244f89d914b8ae95384c18141a7c8cd9"} Nov 01 20:02:57.550368 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:02:57.550347 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-w6g7x" event={"ID":"277af67c-a53b-48c6-9cd0-41c1af1c95dd","Type":"ContainerStarted","Data":"e5e5f01ea3b8af6170e8a648a307d53c622712ad343436d7559bc0515e5871e1"} Nov 01 20:03:00.573746 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:00.573707 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-jbfsn" event={"ID":"33b28b24-2fcc-40d9-9ad0-059f15acf4aa","Type":"ContainerStarted","Data":"1f866d1c51552a2dfdd72c6fb673d718230834b6fa8fa714c636317ca807fd86"} Nov 01 20:03:00.574560 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:00.574533 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-jbfsn" Nov 01 20:03:00.602518 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:00.602475 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-jbfsn" podStartSLOduration=1.78915486 podStartE2EDuration="4.602460961s" totalImagesPullingTime="2.813306101s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:02:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:03:00.601934349 +0000 UTC m=+226.375094866" watchObservedRunningTime="2026-11-01 20:03:00.602460961 +0000 UTC m=+226.375621474" Nov 01 20:03:01.582764 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:01.582720 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-w6g7x" event={"ID":"277af67c-a53b-48c6-9cd0-41c1af1c95dd","Type":"ContainerStarted","Data":"a3211bb491d782de96ff325d7bba5fbf69f7f2a9f582c8c8dac824f6629b93e4"} Nov 01 20:03:01.583220 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:01.582789 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-w6g7x" Nov 01 20:03:01.584192 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:01.584165 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-x9p74" event={"ID":"e4de1911-27fd-43cb-970b-308f8380bd6d","Type":"ContainerStarted","Data":"95e88d652c93c50b8243f397f90225c5e698d5a32d6166a8f4abccf52cba2303"} Nov 01 20:03:01.584336 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:01.584320 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-x9p74" Nov 01 20:03:01.598799 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:01.598758 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-w6g7x" podStartSLOduration=1.725364136 podStartE2EDuration="5.598746498s" totalImagesPullingTime="3.873382362s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:02:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:03:01.597404777 +0000 UTC m=+227.370565306" watchObservedRunningTime="2026-11-01 20:03:01.598746498 +0000 UTC m=+227.371907009" Nov 01 20:03:01.612760 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:01.612719 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-x9p74" podStartSLOduration=1.7173938720000002 podStartE2EDuration="5.612708469s" totalImagesPullingTime="3.895314597s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:02:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:03:01.61226645 +0000 UTC m=+227.385426957" watchObservedRunningTime="2026-11-01 20:03:01.612708469 +0000 UTC m=+227.385868980" Nov 01 20:03:04.533661 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:04.533635 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-jtc4h" Nov 01 20:03:05.638039 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:05.638007 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-jtc4h"] Nov 01 20:03:05.638541 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:05.638269 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-jtc4h" podUID="cc6156d4-83e7-4da4-aa28-3b026753a86c" containerName="limitador" containerID="cri-o://090a0dbc2d9274e417593452decfccc4198b763ec062afd72928feb85133b854" gracePeriod=30 Nov 01 20:03:05.842427 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:03:05.842390 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:03:06.180619 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:06.180593 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-jtc4h" Nov 01 20:03:06.287577 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:06.287500 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc6156d4-83e7-4da4-aa28-3b026753a86c-config-file\" (UniqueName: \"kubernetes.io/configmap/cc6156d4-83e7-4da4-aa28-3b026753a86c-config-file\") pod \"cc6156d4-83e7-4da4-aa28-3b026753a86c\" (UID: \"cc6156d4-83e7-4da4-aa28-3b026753a86c\") " Nov 01 20:03:06.287577 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:06.287548 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc6156d4-83e7-4da4-aa28-3b026753a86c-kube-api-access-j9pb7\" (UniqueName: \"kubernetes.io/projected/cc6156d4-83e7-4da4-aa28-3b026753a86c-kube-api-access-j9pb7\") pod \"cc6156d4-83e7-4da4-aa28-3b026753a86c\" (UID: \"cc6156d4-83e7-4da4-aa28-3b026753a86c\") " Nov 01 20:03:06.287831 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:06.287808 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc6156d4-83e7-4da4-aa28-3b026753a86c-config-file" pod "cc6156d4-83e7-4da4-aa28-3b026753a86c" (UID: "cc6156d4-83e7-4da4-aa28-3b026753a86c"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:03:06.289595 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:06.289570 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc6156d4-83e7-4da4-aa28-3b026753a86c-kube-api-access-j9pb7" pod "cc6156d4-83e7-4da4-aa28-3b026753a86c" (UID: "cc6156d4-83e7-4da4-aa28-3b026753a86c"). InnerVolumeSpecName "kube-api-access-j9pb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:03:06.389241 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:06.389216 5935 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/cc6156d4-83e7-4da4-aa28-3b026753a86c-config-file\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:03:06.389241 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:06.389240 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9pb7\" (UniqueName: \"kubernetes.io/projected/cc6156d4-83e7-4da4-aa28-3b026753a86c-kube-api-access-j9pb7\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:03:06.609042 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:06.608960 5935 generic.go:356] "Generic (PLEG): container finished" podID="cc6156d4-83e7-4da4-aa28-3b026753a86c" containerID="090a0dbc2d9274e417593452decfccc4198b763ec062afd72928feb85133b854" exitCode=0 Nov 01 20:03:06.609042 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:06.609006 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-jtc4h" Nov 01 20:03:06.609234 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:06.609045 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-jtc4h" event={"ID":"cc6156d4-83e7-4da4-aa28-3b026753a86c","Type":"ContainerDied","Data":"090a0dbc2d9274e417593452decfccc4198b763ec062afd72928feb85133b854"} Nov 01 20:03:06.609234 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:06.609096 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-jtc4h" event={"ID":"cc6156d4-83e7-4da4-aa28-3b026753a86c","Type":"ContainerDied","Data":"94b925fc887500b32b0993c510f72d640ca8a9f2f6bcf036272b003891c63401"} Nov 01 20:03:06.609234 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:06.609118 5935 scope.go:122] "RemoveContainer" containerID="090a0dbc2d9274e417593452decfccc4198b763ec062afd72928feb85133b854" Nov 01 20:03:06.617493 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:06.617478 5935 scope.go:122] "RemoveContainer" containerID="090a0dbc2d9274e417593452decfccc4198b763ec062afd72928feb85133b854" Nov 01 20:03:06.617734 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:03:06.617712 5935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"090a0dbc2d9274e417593452decfccc4198b763ec062afd72928feb85133b854\": container with ID starting with 090a0dbc2d9274e417593452decfccc4198b763ec062afd72928feb85133b854 not found: ID does not exist" containerID="090a0dbc2d9274e417593452decfccc4198b763ec062afd72928feb85133b854" Nov 01 20:03:06.617800 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:06.617742 5935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"090a0dbc2d9274e417593452decfccc4198b763ec062afd72928feb85133b854"} err="failed to get container status \"090a0dbc2d9274e417593452decfccc4198b763ec062afd72928feb85133b854\": rpc error: code = NotFound desc = could not find container \"090a0dbc2d9274e417593452decfccc4198b763ec062afd72928feb85133b854\": container with ID starting with 090a0dbc2d9274e417593452decfccc4198b763ec062afd72928feb85133b854 not found: ID does not exist" Nov 01 20:03:06.626323 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:06.626303 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-jtc4h"] Nov 01 20:03:06.631647 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:06.631629 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-jtc4h"] Nov 01 20:03:08.351894 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:08.351863 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc6156d4-83e7-4da4-aa28-3b026753a86c" path="/var/lib/kubelet/pods/cc6156d4-83e7-4da4-aa28-3b026753a86c/volumes" Nov 01 20:03:11.614827 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:11.614795 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-hrbm5"] Nov 01 20:03:11.615189 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:11.615125 5935 cpu_manager.go:414] "containerMap: removing container" podUID="cc6156d4-83e7-4da4-aa28-3b026753a86c" containerName="limitador" Nov 01 20:03:11.615189 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:11.615136 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc6156d4-83e7-4da4-aa28-3b026753a86c" containerName="limitador" Nov 01 20:03:11.615260 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:11.615198 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc6156d4-83e7-4da4-aa28-3b026753a86c" containerName="limitador" Nov 01 20:03:11.619647 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:11.619629 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-hrbm5" Nov 01 20:03:11.628672 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:11.628190 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 01 20:03:11.628672 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:11.628465 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Nov 01 20:03:11.628933 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:11.628912 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 01 20:03:11.629090 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:11.629068 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-8cprv\"" Nov 01 20:03:11.631939 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:11.631907 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-hrbm5"] Nov 01 20:03:11.736026 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:11.735995 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8rxs\" (UniqueName: \"kubernetes.io/projected/69c13cd0-756b-4cc2-9eae-20a1ac83f22d-kube-api-access-n8rxs\") pod \"postgres-7cdbf6dc44-hrbm5\" (UID: \"69c13cd0-756b-4cc2-9eae-20a1ac83f22d\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-hrbm5" Nov 01 20:03:11.736173 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:11.736035 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/69c13cd0-756b-4cc2-9eae-20a1ac83f22d-data\") pod \"postgres-7cdbf6dc44-hrbm5\" (UID: \"69c13cd0-756b-4cc2-9eae-20a1ac83f22d\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-hrbm5" Nov 01 20:03:11.836621 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:11.836584 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8rxs\" (UniqueName: \"kubernetes.io/projected/69c13cd0-756b-4cc2-9eae-20a1ac83f22d-kube-api-access-n8rxs\") pod \"postgres-7cdbf6dc44-hrbm5\" (UID: \"69c13cd0-756b-4cc2-9eae-20a1ac83f22d\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-hrbm5" Nov 01 20:03:11.836779 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:11.836629 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/69c13cd0-756b-4cc2-9eae-20a1ac83f22d-data\") pod \"postgres-7cdbf6dc44-hrbm5\" (UID: \"69c13cd0-756b-4cc2-9eae-20a1ac83f22d\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-hrbm5" Nov 01 20:03:11.836949 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:11.836934 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/69c13cd0-756b-4cc2-9eae-20a1ac83f22d-data\") pod \"postgres-7cdbf6dc44-hrbm5\" (UID: \"69c13cd0-756b-4cc2-9eae-20a1ac83f22d\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-hrbm5" Nov 01 20:03:11.844536 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:11.844514 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8rxs\" (UniqueName: \"kubernetes.io/projected/69c13cd0-756b-4cc2-9eae-20a1ac83f22d-kube-api-access-n8rxs\") pod \"postgres-7cdbf6dc44-hrbm5\" (UID: \"69c13cd0-756b-4cc2-9eae-20a1ac83f22d\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-hrbm5" Nov 01 20:03:11.941661 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:11.941636 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-hrbm5" Nov 01 20:03:12.241421 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:12.241395 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-hrbm5"] Nov 01 20:03:12.243893 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:03:12.243869 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69c13cd0_756b_4cc2_9eae_20a1ac83f22d.slice/crio-93d064d203c4d0bfb4bd303dd85fe6a3ed53902c8017d1445b4fb155d1089233 WatchSource:0}: Error finding container 93d064d203c4d0bfb4bd303dd85fe6a3ed53902c8017d1445b4fb155d1089233: Status 404 returned error can't find the container with id 93d064d203c4d0bfb4bd303dd85fe6a3ed53902c8017d1445b4fb155d1089233 Nov 01 20:03:12.593618 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:12.593543 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-jbfsn" Nov 01 20:03:12.644055 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:12.644017 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-hrbm5" event={"ID":"69c13cd0-756b-4cc2-9eae-20a1ac83f22d","Type":"ContainerStarted","Data":"93d064d203c4d0bfb4bd303dd85fe6a3ed53902c8017d1445b4fb155d1089233"} Nov 01 20:03:17.455238 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:17.455215 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Nov 01 20:03:17.672823 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:17.672785 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-hrbm5" event={"ID":"69c13cd0-756b-4cc2-9eae-20a1ac83f22d","Type":"ContainerStarted","Data":"899ce605980afad7c85111b310c90adce9589f4bd9d0415ad9eba5259f2b4647"} Nov 01 20:03:17.672980 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:17.672908 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-hrbm5" Nov 01 20:03:17.688731 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:17.688663 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-hrbm5" podStartSLOduration=1.481637668 podStartE2EDuration="6.688646278s" totalImagesPullingTime="5.20700861s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:03:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:03:17.687247386 +0000 UTC m=+243.460407898" watchObservedRunningTime="2026-11-01 20:03:17.688646278 +0000 UTC m=+243.461806794" Nov 01 20:03:23.706183 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:23.706159 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-hrbm5" Nov 01 20:03:32.591813 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:32.591779 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-w6g7x" Nov 01 20:03:32.594947 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:32.594921 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-x9p74" Nov 01 20:03:35.843562 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:03:35.843522 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:03:36.730994 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:36.730962 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9hj4l"] Nov 01 20:03:36.740231 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:36.740204 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9hj4l" Nov 01 20:03:36.745431 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:36.745401 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9hj4l"] Nov 01 20:03:36.749998 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:36.749964 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Nov 01 20:03:36.749998 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:36.749958 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Nov 01 20:03:36.751333 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:36.751312 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Nov 01 20:03:36.751441 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:36.751344 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-2gdkn\"" Nov 01 20:03:36.845027 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:36.844995 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/557bbd53-195b-4a00-985e-d3f8647758d1-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-9hj4l\" (UID: \"557bbd53-195b-4a00-985e-d3f8647758d1\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9hj4l" Nov 01 20:03:36.845428 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:36.845029 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/557bbd53-195b-4a00-985e-d3f8647758d1-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-9hj4l\" (UID: \"557bbd53-195b-4a00-985e-d3f8647758d1\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9hj4l" Nov 01 20:03:36.845428 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:36.845112 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/557bbd53-195b-4a00-985e-d3f8647758d1-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-9hj4l\" (UID: \"557bbd53-195b-4a00-985e-d3f8647758d1\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9hj4l" Nov 01 20:03:36.845428 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:36.845220 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bz98\" (UniqueName: \"kubernetes.io/projected/557bbd53-195b-4a00-985e-d3f8647758d1-kube-api-access-8bz98\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-9hj4l\" (UID: \"557bbd53-195b-4a00-985e-d3f8647758d1\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9hj4l" Nov 01 20:03:36.946623 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:36.946592 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bz98\" (UniqueName: \"kubernetes.io/projected/557bbd53-195b-4a00-985e-d3f8647758d1-kube-api-access-8bz98\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-9hj4l\" (UID: \"557bbd53-195b-4a00-985e-d3f8647758d1\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9hj4l" Nov 01 20:03:36.946787 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:36.946642 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/557bbd53-195b-4a00-985e-d3f8647758d1-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-9hj4l\" (UID: \"557bbd53-195b-4a00-985e-d3f8647758d1\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9hj4l" Nov 01 20:03:36.946787 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:36.946755 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/557bbd53-195b-4a00-985e-d3f8647758d1-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-9hj4l\" (UID: \"557bbd53-195b-4a00-985e-d3f8647758d1\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9hj4l" Nov 01 20:03:36.946896 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:36.946809 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/557bbd53-195b-4a00-985e-d3f8647758d1-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-9hj4l\" (UID: \"557bbd53-195b-4a00-985e-d3f8647758d1\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9hj4l" Nov 01 20:03:36.947439 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:36.947417 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/557bbd53-195b-4a00-985e-d3f8647758d1-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-9hj4l\" (UID: \"557bbd53-195b-4a00-985e-d3f8647758d1\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9hj4l" Nov 01 20:03:36.947610 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:36.947591 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/557bbd53-195b-4a00-985e-d3f8647758d1-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-9hj4l\" (UID: \"557bbd53-195b-4a00-985e-d3f8647758d1\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9hj4l" Nov 01 20:03:36.949170 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:36.949150 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/557bbd53-195b-4a00-985e-d3f8647758d1-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-9hj4l\" (UID: \"557bbd53-195b-4a00-985e-d3f8647758d1\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9hj4l" Nov 01 20:03:36.954491 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:36.954473 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bz98\" (UniqueName: \"kubernetes.io/projected/557bbd53-195b-4a00-985e-d3f8647758d1-kube-api-access-8bz98\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-9hj4l\" (UID: \"557bbd53-195b-4a00-985e-d3f8647758d1\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9hj4l" Nov 01 20:03:37.056441 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:37.056377 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9hj4l" Nov 01 20:03:37.323406 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:37.323384 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9hj4l"] Nov 01 20:03:37.325877 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:03:37.325849 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod557bbd53_195b_4a00_985e_d3f8647758d1.slice/crio-f627659292062834f6a715ebaac8901328484068be9c79772a43296a9e4d9477 WatchSource:0}: Error finding container f627659292062834f6a715ebaac8901328484068be9c79772a43296a9e4d9477: Status 404 returned error can't find the container with id f627659292062834f6a715ebaac8901328484068be9c79772a43296a9e4d9477 Nov 01 20:03:37.766199 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:37.766169 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9hj4l" event={"ID":"557bbd53-195b-4a00-985e-d3f8647758d1","Type":"ContainerStarted","Data":"f627659292062834f6a715ebaac8901328484068be9c79772a43296a9e4d9477"} Nov 01 20:03:49.837191 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:49.837152 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9hj4l" event={"ID":"557bbd53-195b-4a00-985e-d3f8647758d1","Type":"ContainerStarted","Data":"4a04edea516fda1328305df843fd1ecb28c35320e53c4c9343bc058812203575"} Nov 01 20:03:49.837608 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:49.837322 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9hj4l" Nov 01 20:03:49.854552 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:03:49.854450 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9hj4l" podStartSLOduration=1.5867906299999999 podStartE2EDuration="13.854433225s" totalImagesPullingTime="12.267642595s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:03:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:03:49.854348341 +0000 UTC m=+275.627508854" watchObservedRunningTime="2026-11-01 20:03:49.854433225 +0000 UTC m=+275.627593772" Nov 01 20:04:00.847380 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:00.847354 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-9hj4l" Nov 01 20:04:05.843070 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:04:05.843025 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:04:07.591079 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:07.591051 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5fc46f9f99-2bth4"] Nov 01 20:04:07.594636 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:07.594616 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5fc46f9f99-2bth4" Nov 01 20:04:07.600359 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:07.600339 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 01 20:04:07.601497 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:07.601474 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7vx47\"" Nov 01 20:04:07.603815 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:07.603793 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 01 20:04:07.618648 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:07.618624 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5fc46f9f99-2bth4"] Nov 01 20:04:07.727377 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:07.727342 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9b2m\" (UniqueName: \"kubernetes.io/projected/40b947b8-e5e4-4742-ab33-76f6f15f4978-kube-api-access-r9b2m\") pod \"maas-api-5fc46f9f99-2bth4\" (UID: \"40b947b8-e5e4-4742-ab33-76f6f15f4978\") " pod="odh-ai-gateway-infra/maas-api-5fc46f9f99-2bth4" Nov 01 20:04:07.727569 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:07.727390 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40b947b8-e5e4-4742-ab33-76f6f15f4978-metrics-certs\") pod \"maas-api-5fc46f9f99-2bth4\" (UID: \"40b947b8-e5e4-4742-ab33-76f6f15f4978\") " pod="odh-ai-gateway-infra/maas-api-5fc46f9f99-2bth4" Nov 01 20:04:07.727569 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:07.727413 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40b947b8-e5e4-4742-ab33-76f6f15f4978-openshift-service-ca\") pod \"maas-api-5fc46f9f99-2bth4\" (UID: \"40b947b8-e5e4-4742-ab33-76f6f15f4978\") " pod="odh-ai-gateway-infra/maas-api-5fc46f9f99-2bth4" Nov 01 20:04:07.727695 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:07.727586 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40b947b8-e5e4-4742-ab33-76f6f15f4978-maas-api-tls\") pod \"maas-api-5fc46f9f99-2bth4\" (UID: \"40b947b8-e5e4-4742-ab33-76f6f15f4978\") " pod="odh-ai-gateway-infra/maas-api-5fc46f9f99-2bth4" Nov 01 20:04:07.828396 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:07.828358 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40b947b8-e5e4-4742-ab33-76f6f15f4978-metrics-certs\") pod \"maas-api-5fc46f9f99-2bth4\" (UID: \"40b947b8-e5e4-4742-ab33-76f6f15f4978\") " pod="odh-ai-gateway-infra/maas-api-5fc46f9f99-2bth4" Nov 01 20:04:07.828396 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:07.828400 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40b947b8-e5e4-4742-ab33-76f6f15f4978-openshift-service-ca\") pod \"maas-api-5fc46f9f99-2bth4\" (UID: \"40b947b8-e5e4-4742-ab33-76f6f15f4978\") " pod="odh-ai-gateway-infra/maas-api-5fc46f9f99-2bth4" Nov 01 20:04:07.828606 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:07.828439 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40b947b8-e5e4-4742-ab33-76f6f15f4978-maas-api-tls\") pod \"maas-api-5fc46f9f99-2bth4\" (UID: \"40b947b8-e5e4-4742-ab33-76f6f15f4978\") " pod="odh-ai-gateway-infra/maas-api-5fc46f9f99-2bth4" Nov 01 20:04:07.828606 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:07.828555 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9b2m\" (UniqueName: \"kubernetes.io/projected/40b947b8-e5e4-4742-ab33-76f6f15f4978-kube-api-access-r9b2m\") pod \"maas-api-5fc46f9f99-2bth4\" (UID: \"40b947b8-e5e4-4742-ab33-76f6f15f4978\") " pod="odh-ai-gateway-infra/maas-api-5fc46f9f99-2bth4" Nov 01 20:04:07.829169 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:07.829146 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40b947b8-e5e4-4742-ab33-76f6f15f4978-openshift-service-ca\") pod \"maas-api-5fc46f9f99-2bth4\" (UID: \"40b947b8-e5e4-4742-ab33-76f6f15f4978\") " pod="odh-ai-gateway-infra/maas-api-5fc46f9f99-2bth4" Nov 01 20:04:07.830790 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:07.830769 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40b947b8-e5e4-4742-ab33-76f6f15f4978-metrics-certs\") pod \"maas-api-5fc46f9f99-2bth4\" (UID: \"40b947b8-e5e4-4742-ab33-76f6f15f4978\") " pod="odh-ai-gateway-infra/maas-api-5fc46f9f99-2bth4" Nov 01 20:04:07.830890 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:07.830815 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40b947b8-e5e4-4742-ab33-76f6f15f4978-maas-api-tls\") pod \"maas-api-5fc46f9f99-2bth4\" (UID: \"40b947b8-e5e4-4742-ab33-76f6f15f4978\") " pod="odh-ai-gateway-infra/maas-api-5fc46f9f99-2bth4" Nov 01 20:04:07.836495 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:07.836476 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9b2m\" (UniqueName: \"kubernetes.io/projected/40b947b8-e5e4-4742-ab33-76f6f15f4978-kube-api-access-r9b2m\") pod \"maas-api-5fc46f9f99-2bth4\" (UID: \"40b947b8-e5e4-4742-ab33-76f6f15f4978\") " pod="odh-ai-gateway-infra/maas-api-5fc46f9f99-2bth4" Nov 01 20:04:07.906983 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:07.906906 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5fc46f9f99-2bth4" Nov 01 20:04:08.184565 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:08.184539 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5fc46f9f99-2bth4"] Nov 01 20:04:08.187092 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:04:08.187060 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40b947b8_e5e4_4742_ab33_76f6f15f4978.slice/crio-7899549eacc9cb1a29f9c74772bfbaa132028f794f9d87b19d680dd7f2a8f2e8 WatchSource:0}: Error finding container 7899549eacc9cb1a29f9c74772bfbaa132028f794f9d87b19d680dd7f2a8f2e8: Status 404 returned error can't find the container with id 7899549eacc9cb1a29f9c74772bfbaa132028f794f9d87b19d680dd7f2a8f2e8 Nov 01 20:04:08.928213 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:08.928180 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5fc46f9f99-2bth4" event={"ID":"40b947b8-e5e4-4742-ab33-76f6f15f4978","Type":"ContainerStarted","Data":"7899549eacc9cb1a29f9c74772bfbaa132028f794f9d87b19d680dd7f2a8f2e8"} Nov 01 20:04:12.976702 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:12.976666 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5fc46f9f99-2bth4" event={"ID":"40b947b8-e5e4-4742-ab33-76f6f15f4978","Type":"ContainerStarted","Data":"07fcadb80c898ec3b06a6bc7b81eeb84b805bba0f434662c12d3bb62091659a7"} Nov 01 20:04:12.977103 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:12.976776 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5fc46f9f99-2bth4" Nov 01 20:04:12.994597 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:12.994550 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5fc46f9f99-2bth4" podStartSLOduration=1.656988804 podStartE2EDuration="5.99453476s" totalImagesPullingTime="4.337545956s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:04:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:04:12.992082423 +0000 UTC m=+298.765242936" watchObservedRunningTime="2026-11-01 20:04:12.99453476 +0000 UTC m=+298.767695272" Nov 01 20:04:18.592495 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:18.587700 5935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-ljtcq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": read tcp 10.133.0.2:43226->10.133.0.22:15021: read: connection reset by peer" start-of-body= Nov 01 20:04:18.592495 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:18.587764 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-ljtcq" podUID="03014616-4520-48bb-a548-66485577e26c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": read tcp 10.133.0.2:43226->10.133.0.22:15021: read: connection reset by peer" Nov 01 20:04:18.987018 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:18.986990 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5fc46f9f99-2bth4" Nov 01 20:04:19.007933 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:19.007903 5935 generic.go:356] "Generic (PLEG): container finished" podID="03014616-4520-48bb-a548-66485577e26c" containerID="76c8578d02abb6f5d2c0a60721d1fcc733fbed26f7bfb19d7efa857a4c1a5df4" exitCode=0 Nov 01 20:04:19.008072 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:19.007940 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-ljtcq" event={"ID":"03014616-4520-48bb-a548-66485577e26c","Type":"ContainerDied","Data":"76c8578d02abb6f5d2c0a60721d1fcc733fbed26f7bfb19d7efa857a4c1a5df4"} Nov 01 20:04:19.008332 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:19.008315 5935 scope.go:122] "RemoveContainer" containerID="76c8578d02abb6f5d2c0a60721d1fcc733fbed26f7bfb19d7efa857a4c1a5df4" Nov 01 20:04:19.009363 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:19.009335 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:04:19.009422 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:19.009398 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:04:19.009460 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:19.009433 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:04:20.015724 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:20.015688 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-ljtcq" event={"ID":"03014616-4520-48bb-a548-66485577e26c","Type":"ContainerStarted","Data":"4ec976c9f232fb978717e396f792859cc232a3a6391b7580a66809ad3a7c6421"} Nov 01 20:04:20.662951 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:20.659681 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-ljtcq" Nov 01 20:04:20.662951 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:20.659813 5935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-ljtcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" start-of-body= Nov 01 20:04:20.662951 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:20.659857 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-ljtcq" podUID="03014616-4520-48bb-a548-66485577e26c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" Nov 01 20:04:21.667312 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:21.665717 5935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-ljtcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" start-of-body= Nov 01 20:04:21.667312 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:21.665789 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-ljtcq" podUID="03014616-4520-48bb-a548-66485577e26c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" Nov 01 20:04:22.667852 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:22.667814 5935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-ljtcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" start-of-body= Nov 01 20:04:22.668393 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:22.667881 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-ljtcq" podUID="03014616-4520-48bb-a548-66485577e26c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" Nov 01 20:04:23.663672 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:23.663637 5935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-ljtcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" start-of-body= Nov 01 20:04:23.663947 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:23.663918 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-ljtcq" podUID="03014616-4520-48bb-a548-66485577e26c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" Nov 01 20:04:24.267449 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:24.267416 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-vtpsz"] Nov 01 20:04:24.283225 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:24.283198 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-vtpsz"] Nov 01 20:04:24.283386 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:24.283339 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-vtpsz" Nov 01 20:04:24.302956 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:24.302930 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r99v\" (UniqueName: \"kubernetes.io/projected/458588d6-f7a6-4e19-a2e5-75bf2a9373fc-kube-api-access-4r99v\") pod \"llmisvc-controller-manager-6466c554d8-vtpsz\" (UID: \"458588d6-f7a6-4e19-a2e5-75bf2a9373fc\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-vtpsz" Nov 01 20:04:24.303070 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:24.302996 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/458588d6-f7a6-4e19-a2e5-75bf2a9373fc-cert\") pod \"llmisvc-controller-manager-6466c554d8-vtpsz\" (UID: \"458588d6-f7a6-4e19-a2e5-75bf2a9373fc\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-vtpsz" Nov 01 20:04:24.404352 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:24.404322 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r99v\" (UniqueName: \"kubernetes.io/projected/458588d6-f7a6-4e19-a2e5-75bf2a9373fc-kube-api-access-4r99v\") pod \"llmisvc-controller-manager-6466c554d8-vtpsz\" (UID: \"458588d6-f7a6-4e19-a2e5-75bf2a9373fc\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-vtpsz" Nov 01 20:04:24.404537 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:24.404495 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/458588d6-f7a6-4e19-a2e5-75bf2a9373fc-cert\") pod \"llmisvc-controller-manager-6466c554d8-vtpsz\" (UID: \"458588d6-f7a6-4e19-a2e5-75bf2a9373fc\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-vtpsz" Nov 01 20:04:24.407175 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:24.407152 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/458588d6-f7a6-4e19-a2e5-75bf2a9373fc-cert\") pod \"llmisvc-controller-manager-6466c554d8-vtpsz\" (UID: \"458588d6-f7a6-4e19-a2e5-75bf2a9373fc\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-vtpsz" Nov 01 20:04:24.412903 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:24.412877 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r99v\" (UniqueName: \"kubernetes.io/projected/458588d6-f7a6-4e19-a2e5-75bf2a9373fc-kube-api-access-4r99v\") pod \"llmisvc-controller-manager-6466c554d8-vtpsz\" (UID: \"458588d6-f7a6-4e19-a2e5-75bf2a9373fc\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-vtpsz" Nov 01 20:04:24.596457 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:24.596379 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-vtpsz" Nov 01 20:04:24.669603 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:24.659675 5935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-ljtcq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" start-of-body= Nov 01 20:04:24.669603 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:24.659743 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-ljtcq" podUID="03014616-4520-48bb-a548-66485577e26c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": dial tcp 10.133.0.22:15021: connect: connection refused" Nov 01 20:04:24.886772 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:24.886742 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-vtpsz"] Nov 01 20:04:24.889686 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:04:24.889651 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod458588d6_f7a6_4e19_a2e5_75bf2a9373fc.slice/crio-c2d58528475d9a2fc86ceec51ad07b5823c80ccb23e00316d8900e77cc206fcb WatchSource:0}: Error finding container c2d58528475d9a2fc86ceec51ad07b5823c80ccb23e00316d8900e77cc206fcb: Status 404 returned error can't find the container with id c2d58528475d9a2fc86ceec51ad07b5823c80ccb23e00316d8900e77cc206fcb Nov 01 20:04:25.039554 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:25.039520 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-vtpsz" event={"ID":"458588d6-f7a6-4e19-a2e5-75bf2a9373fc","Type":"ContainerStarted","Data":"c2d58528475d9a2fc86ceec51ad07b5823c80ccb23e00316d8900e77cc206fcb"} Nov 01 20:04:25.662134 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:25.662063 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-ljtcq" Nov 01 20:04:25.662526 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:25.662323 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-ljtcq" Nov 01 20:04:25.663139 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:25.663119 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-ljtcq" Nov 01 20:04:26.045377 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:26.045344 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-vtpsz" event={"ID":"458588d6-f7a6-4e19-a2e5-75bf2a9373fc","Type":"ContainerStarted","Data":"48a4b556f3a48321be173647e32ae264bba1e45e5cffe0f5c80ff3bf01e8c243"} Nov 01 20:04:26.045555 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:26.045411 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-vtpsz" Nov 01 20:04:26.062109 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:26.062055 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-vtpsz" podStartSLOduration=1.705238251 podStartE2EDuration="2.062037876s" totalImagesPullingTime="356.799625ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:04:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:04:26.06173132 +0000 UTC m=+311.834891831" watchObservedRunningTime="2026-11-01 20:04:26.062037876 +0000 UTC m=+311.835198391" Nov 01 20:04:29.034715 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:29.034676 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5fc46f9f99-2bth4"] Nov 01 20:04:29.035175 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:29.034922 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5fc46f9f99-2bth4" podUID="40b947b8-e5e4-4742-ab33-76f6f15f4978" containerName="maas-api" containerID="cri-o://07fcadb80c898ec3b06a6bc7b81eeb84b805bba0f434662c12d3bb62091659a7" gracePeriod=30 Nov 01 20:04:29.275803 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:29.275781 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5fc46f9f99-2bth4" Nov 01 20:04:29.347608 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:29.347552 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40b947b8-e5e4-4742-ab33-76f6f15f4978-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40b947b8-e5e4-4742-ab33-76f6f15f4978-openshift-service-ca\") pod \"40b947b8-e5e4-4742-ab33-76f6f15f4978\" (UID: \"40b947b8-e5e4-4742-ab33-76f6f15f4978\") " Nov 01 20:04:29.347608 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:29.347586 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40b947b8-e5e4-4742-ab33-76f6f15f4978-kube-api-access-r9b2m\" (UniqueName: \"kubernetes.io/projected/40b947b8-e5e4-4742-ab33-76f6f15f4978-kube-api-access-r9b2m\") pod \"40b947b8-e5e4-4742-ab33-76f6f15f4978\" (UID: \"40b947b8-e5e4-4742-ab33-76f6f15f4978\") " Nov 01 20:04:29.347608 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:29.347605 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40b947b8-e5e4-4742-ab33-76f6f15f4978-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40b947b8-e5e4-4742-ab33-76f6f15f4978-maas-api-tls\") pod \"40b947b8-e5e4-4742-ab33-76f6f15f4978\" (UID: \"40b947b8-e5e4-4742-ab33-76f6f15f4978\") " Nov 01 20:04:29.347855 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:29.347628 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40b947b8-e5e4-4742-ab33-76f6f15f4978-metrics-certs\" (UniqueName: \"kubernetes.io/secret/40b947b8-e5e4-4742-ab33-76f6f15f4978-metrics-certs\") pod \"40b947b8-e5e4-4742-ab33-76f6f15f4978\" (UID: \"40b947b8-e5e4-4742-ab33-76f6f15f4978\") " Nov 01 20:04:29.347919 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:29.347879 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40b947b8-e5e4-4742-ab33-76f6f15f4978-openshift-service-ca" pod "40b947b8-e5e4-4742-ab33-76f6f15f4978" (UID: "40b947b8-e5e4-4742-ab33-76f6f15f4978"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:04:29.349783 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:29.349743 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40b947b8-e5e4-4742-ab33-76f6f15f4978-kube-api-access-r9b2m" pod "40b947b8-e5e4-4742-ab33-76f6f15f4978" (UID: "40b947b8-e5e4-4742-ab33-76f6f15f4978"). InnerVolumeSpecName "kube-api-access-r9b2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:04:29.349783 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:29.349755 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40b947b8-e5e4-4742-ab33-76f6f15f4978-maas-api-tls" pod "40b947b8-e5e4-4742-ab33-76f6f15f4978" (UID: "40b947b8-e5e4-4742-ab33-76f6f15f4978"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:04:29.349915 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:29.349791 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40b947b8-e5e4-4742-ab33-76f6f15f4978-metrics-certs" pod "40b947b8-e5e4-4742-ab33-76f6f15f4978" (UID: "40b947b8-e5e4-4742-ab33-76f6f15f4978"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:04:29.448679 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:29.448645 5935 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40b947b8-e5e4-4742-ab33-76f6f15f4978-openshift-service-ca\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:04:29.448679 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:29.448673 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9b2m\" (UniqueName: \"kubernetes.io/projected/40b947b8-e5e4-4742-ab33-76f6f15f4978-kube-api-access-r9b2m\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:04:29.448848 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:29.448688 5935 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40b947b8-e5e4-4742-ab33-76f6f15f4978-maas-api-tls\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:04:29.448848 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:29.448700 5935 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40b947b8-e5e4-4742-ab33-76f6f15f4978-metrics-certs\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:04:30.065012 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:30.064981 5935 generic.go:356] "Generic (PLEG): container finished" podID="40b947b8-e5e4-4742-ab33-76f6f15f4978" containerID="07fcadb80c898ec3b06a6bc7b81eeb84b805bba0f434662c12d3bb62091659a7" exitCode=0 Nov 01 20:04:30.065446 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:30.065015 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5fc46f9f99-2bth4" event={"ID":"40b947b8-e5e4-4742-ab33-76f6f15f4978","Type":"ContainerDied","Data":"07fcadb80c898ec3b06a6bc7b81eeb84b805bba0f434662c12d3bb62091659a7"} Nov 01 20:04:30.065446 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:30.065022 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5fc46f9f99-2bth4" Nov 01 20:04:30.065446 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:30.065059 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5fc46f9f99-2bth4" event={"ID":"40b947b8-e5e4-4742-ab33-76f6f15f4978","Type":"ContainerDied","Data":"7899549eacc9cb1a29f9c74772bfbaa132028f794f9d87b19d680dd7f2a8f2e8"} Nov 01 20:04:30.065446 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:30.065080 5935 scope.go:122] "RemoveContainer" containerID="07fcadb80c898ec3b06a6bc7b81eeb84b805bba0f434662c12d3bb62091659a7" Nov 01 20:04:30.075106 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:30.075086 5935 scope.go:122] "RemoveContainer" containerID="07fcadb80c898ec3b06a6bc7b81eeb84b805bba0f434662c12d3bb62091659a7" Nov 01 20:04:30.075377 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:04:30.075358 5935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07fcadb80c898ec3b06a6bc7b81eeb84b805bba0f434662c12d3bb62091659a7\": container with ID starting with 07fcadb80c898ec3b06a6bc7b81eeb84b805bba0f434662c12d3bb62091659a7 not found: ID does not exist" containerID="07fcadb80c898ec3b06a6bc7b81eeb84b805bba0f434662c12d3bb62091659a7" Nov 01 20:04:30.075439 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:30.075385 5935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07fcadb80c898ec3b06a6bc7b81eeb84b805bba0f434662c12d3bb62091659a7"} err="failed to get container status \"07fcadb80c898ec3b06a6bc7b81eeb84b805bba0f434662c12d3bb62091659a7\": rpc error: code = NotFound desc = could not find container \"07fcadb80c898ec3b06a6bc7b81eeb84b805bba0f434662c12d3bb62091659a7\": container with ID starting with 07fcadb80c898ec3b06a6bc7b81eeb84b805bba0f434662c12d3bb62091659a7 not found: ID does not exist" Nov 01 20:04:30.087256 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:30.087227 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5fc46f9f99-2bth4"] Nov 01 20:04:30.092840 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:30.092819 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5fc46f9f99-2bth4"] Nov 01 20:04:30.352218 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:30.352142 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40b947b8-e5e4-4742-ab33-76f6f15f4978" path="/var/lib/kubelet/pods/40b947b8-e5e4-4742-ab33-76f6f15f4978/volumes" Nov 01 20:04:35.842438 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:04:35.842398 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:04:57.052135 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:57.052107 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-vtpsz" Nov 01 20:04:57.091889 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:57.091861 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-w6g7x"] Nov 01 20:04:57.092302 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:57.092246 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-w6g7x" podUID="277af67c-a53b-48c6-9cd0-41c1af1c95dd" containerName="manager" containerID="cri-o://a3211bb491d782de96ff325d7bba5fbf69f7f2a9f582c8c8dac824f6629b93e4" gracePeriod=30 Nov 01 20:04:57.343420 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:57.343394 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-w6g7x" Nov 01 20:04:57.474172 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:57.474143 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/277af67c-a53b-48c6-9cd0-41c1af1c95dd-cert\" (UniqueName: \"kubernetes.io/secret/277af67c-a53b-48c6-9cd0-41c1af1c95dd-cert\") pod \"277af67c-a53b-48c6-9cd0-41c1af1c95dd\" (UID: \"277af67c-a53b-48c6-9cd0-41c1af1c95dd\") " Nov 01 20:04:57.474362 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:57.474178 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/277af67c-a53b-48c6-9cd0-41c1af1c95dd-kube-api-access-q94j6\" (UniqueName: \"kubernetes.io/projected/277af67c-a53b-48c6-9cd0-41c1af1c95dd-kube-api-access-q94j6\") pod \"277af67c-a53b-48c6-9cd0-41c1af1c95dd\" (UID: \"277af67c-a53b-48c6-9cd0-41c1af1c95dd\") " Nov 01 20:04:57.476320 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:57.476274 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/277af67c-a53b-48c6-9cd0-41c1af1c95dd-cert" pod "277af67c-a53b-48c6-9cd0-41c1af1c95dd" (UID: "277af67c-a53b-48c6-9cd0-41c1af1c95dd"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:04:57.476438 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:57.476350 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/277af67c-a53b-48c6-9cd0-41c1af1c95dd-kube-api-access-q94j6" pod "277af67c-a53b-48c6-9cd0-41c1af1c95dd" (UID: "277af67c-a53b-48c6-9cd0-41c1af1c95dd"). InnerVolumeSpecName "kube-api-access-q94j6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:04:57.575742 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:57.575671 5935 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/277af67c-a53b-48c6-9cd0-41c1af1c95dd-cert\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:04:57.575742 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:57.575697 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q94j6\" (UniqueName: \"kubernetes.io/projected/277af67c-a53b-48c6-9cd0-41c1af1c95dd-kube-api-access-q94j6\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:04:58.202515 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:58.202479 5935 generic.go:356] "Generic (PLEG): container finished" podID="277af67c-a53b-48c6-9cd0-41c1af1c95dd" containerID="a3211bb491d782de96ff325d7bba5fbf69f7f2a9f582c8c8dac824f6629b93e4" exitCode=0 Nov 01 20:04:58.202961 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:58.202527 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-w6g7x" Nov 01 20:04:58.202961 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:58.202559 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-w6g7x" event={"ID":"277af67c-a53b-48c6-9cd0-41c1af1c95dd","Type":"ContainerDied","Data":"a3211bb491d782de96ff325d7bba5fbf69f7f2a9f582c8c8dac824f6629b93e4"} Nov 01 20:04:58.202961 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:58.202614 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-w6g7x" event={"ID":"277af67c-a53b-48c6-9cd0-41c1af1c95dd","Type":"ContainerDied","Data":"e5e5f01ea3b8af6170e8a648a307d53c622712ad343436d7559bc0515e5871e1"} Nov 01 20:04:58.202961 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:58.202636 5935 scope.go:122] "RemoveContainer" containerID="a3211bb491d782de96ff325d7bba5fbf69f7f2a9f582c8c8dac824f6629b93e4" Nov 01 20:04:58.212591 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:58.212577 5935 scope.go:122] "RemoveContainer" containerID="a3211bb491d782de96ff325d7bba5fbf69f7f2a9f582c8c8dac824f6629b93e4" Nov 01 20:04:58.212862 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:04:58.212840 5935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3211bb491d782de96ff325d7bba5fbf69f7f2a9f582c8c8dac824f6629b93e4\": container with ID starting with a3211bb491d782de96ff325d7bba5fbf69f7f2a9f582c8c8dac824f6629b93e4 not found: ID does not exist" containerID="a3211bb491d782de96ff325d7bba5fbf69f7f2a9f582c8c8dac824f6629b93e4" Nov 01 20:04:58.212938 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:58.212870 5935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3211bb491d782de96ff325d7bba5fbf69f7f2a9f582c8c8dac824f6629b93e4"} err="failed to get container status \"a3211bb491d782de96ff325d7bba5fbf69f7f2a9f582c8c8dac824f6629b93e4\": rpc error: code = NotFound desc = could not find container \"a3211bb491d782de96ff325d7bba5fbf69f7f2a9f582c8c8dac824f6629b93e4\": container with ID starting with a3211bb491d782de96ff325d7bba5fbf69f7f2a9f582c8c8dac824f6629b93e4 not found: ID does not exist" Nov 01 20:04:58.223168 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:58.223143 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-w6g7x"] Nov 01 20:04:58.226630 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:58.226609 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-w6g7x"] Nov 01 20:04:58.353020 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:04:58.352979 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="277af67c-a53b-48c6-9cd0-41c1af1c95dd" path="/var/lib/kubelet/pods/277af67c-a53b-48c6-9cd0-41c1af1c95dd/volumes" Nov 01 20:05:05.843506 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:05:05.843466 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:05:35.845231 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:05:35.845193 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:06:05.843567 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:06:05.843531 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:06:22.723377 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:22.723346 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6qv8"] Nov 01 20:06:22.724176 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:22.723853 5935 cpu_manager.go:414] "containerMap: removing container" podUID="277af67c-a53b-48c6-9cd0-41c1af1c95dd" containerName="manager" Nov 01 20:06:22.724176 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:22.723877 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="277af67c-a53b-48c6-9cd0-41c1af1c95dd" containerName="manager" Nov 01 20:06:22.724176 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:22.723892 5935 cpu_manager.go:414] "containerMap: removing container" podUID="40b947b8-e5e4-4742-ab33-76f6f15f4978" containerName="maas-api" Nov 01 20:06:22.724176 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:22.723900 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40b947b8-e5e4-4742-ab33-76f6f15f4978" containerName="maas-api" Nov 01 20:06:22.724176 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:22.724009 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="277af67c-a53b-48c6-9cd0-41c1af1c95dd" containerName="manager" Nov 01 20:06:22.724176 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:22.724024 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="40b947b8-e5e4-4742-ab33-76f6f15f4978" containerName="maas-api" Nov 01 20:06:22.727394 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:22.727374 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6qv8" Nov 01 20:06:22.742660 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:22.742633 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6qv8"] Nov 01 20:06:22.807427 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:22.807393 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de91381-a29d-422d-b859-69229e8970c7-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-m6qv8\" (UID: \"9de91381-a29d-422d-b859-69229e8970c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6qv8" Nov 01 20:06:22.807594 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:22.807440 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9de91381-a29d-422d-b859-69229e8970c7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-m6qv8\" (UID: \"9de91381-a29d-422d-b859-69229e8970c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6qv8" Nov 01 20:06:22.807594 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:22.807470 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9de91381-a29d-422d-b859-69229e8970c7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-m6qv8\" (UID: \"9de91381-a29d-422d-b859-69229e8970c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6qv8" Nov 01 20:06:22.807672 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:22.807586 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de91381-a29d-422d-b859-69229e8970c7-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-m6qv8\" (UID: \"9de91381-a29d-422d-b859-69229e8970c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6qv8" Nov 01 20:06:22.807672 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:22.807613 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9de91381-a29d-422d-b859-69229e8970c7-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-m6qv8\" (UID: \"9de91381-a29d-422d-b859-69229e8970c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6qv8" Nov 01 20:06:22.807672 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:22.807632 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrhdv\" (UniqueName: \"kubernetes.io/projected/9de91381-a29d-422d-b859-69229e8970c7-kube-api-access-vrhdv\") pod \"maas-default-gateway-openshift-default-fd4b99879-m6qv8\" (UID: \"9de91381-a29d-422d-b859-69229e8970c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6qv8" Nov 01 20:06:22.807766 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:22.807736 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9de91381-a29d-422d-b859-69229e8970c7-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-m6qv8\" (UID: \"9de91381-a29d-422d-b859-69229e8970c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6qv8" Nov 01 20:06:22.807802 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:22.807773 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9de91381-a29d-422d-b859-69229e8970c7-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-m6qv8\" (UID: \"9de91381-a29d-422d-b859-69229e8970c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6qv8" Nov 01 20:06:22.807903 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:22.807882 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9de91381-a29d-422d-b859-69229e8970c7-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-m6qv8\" (UID: \"9de91381-a29d-422d-b859-69229e8970c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6qv8" Nov 01 20:06:22.909206 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:22.909174 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de91381-a29d-422d-b859-69229e8970c7-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-m6qv8\" (UID: \"9de91381-a29d-422d-b859-69229e8970c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6qv8" Nov 01 20:06:22.909206 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:22.909208 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9de91381-a29d-422d-b859-69229e8970c7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-m6qv8\" (UID: \"9de91381-a29d-422d-b859-69229e8970c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6qv8" Nov 01 20:06:22.909455 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:22.909227 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9de91381-a29d-422d-b859-69229e8970c7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-m6qv8\" (UID: \"9de91381-a29d-422d-b859-69229e8970c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6qv8" Nov 01 20:06:22.909455 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:22.909252 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de91381-a29d-422d-b859-69229e8970c7-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-m6qv8\" (UID: \"9de91381-a29d-422d-b859-69229e8970c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6qv8" Nov 01 20:06:22.909455 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:22.909403 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9de91381-a29d-422d-b859-69229e8970c7-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-m6qv8\" (UID: \"9de91381-a29d-422d-b859-69229e8970c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6qv8" Nov 01 20:06:22.909455 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:22.909441 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrhdv\" (UniqueName: \"kubernetes.io/projected/9de91381-a29d-422d-b859-69229e8970c7-kube-api-access-vrhdv\") pod \"maas-default-gateway-openshift-default-fd4b99879-m6qv8\" (UID: \"9de91381-a29d-422d-b859-69229e8970c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6qv8" Nov 01 20:06:22.909669 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:22.909499 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9de91381-a29d-422d-b859-69229e8970c7-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-m6qv8\" (UID: \"9de91381-a29d-422d-b859-69229e8970c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6qv8" Nov 01 20:06:22.909669 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:22.909545 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9de91381-a29d-422d-b859-69229e8970c7-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-m6qv8\" (UID: \"9de91381-a29d-422d-b859-69229e8970c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6qv8" Nov 01 20:06:22.909669 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:22.909623 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9de91381-a29d-422d-b859-69229e8970c7-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-m6qv8\" (UID: \"9de91381-a29d-422d-b859-69229e8970c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6qv8" Nov 01 20:06:22.909669 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:22.909619 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de91381-a29d-422d-b859-69229e8970c7-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-m6qv8\" (UID: \"9de91381-a29d-422d-b859-69229e8970c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6qv8" Nov 01 20:06:22.909669 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:22.909667 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9de91381-a29d-422d-b859-69229e8970c7-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-m6qv8\" (UID: \"9de91381-a29d-422d-b859-69229e8970c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6qv8" Nov 01 20:06:22.909951 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:22.909910 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9de91381-a29d-422d-b859-69229e8970c7-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-m6qv8\" (UID: \"9de91381-a29d-422d-b859-69229e8970c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6qv8" Nov 01 20:06:22.910008 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:22.909954 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9de91381-a29d-422d-b859-69229e8970c7-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-m6qv8\" (UID: \"9de91381-a29d-422d-b859-69229e8970c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6qv8" Nov 01 20:06:22.910060 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:22.910018 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9de91381-a29d-422d-b859-69229e8970c7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-m6qv8\" (UID: \"9de91381-a29d-422d-b859-69229e8970c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6qv8" Nov 01 20:06:22.911705 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:22.911684 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9de91381-a29d-422d-b859-69229e8970c7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-m6qv8\" (UID: \"9de91381-a29d-422d-b859-69229e8970c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6qv8" Nov 01 20:06:22.911954 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:22.911937 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9de91381-a29d-422d-b859-69229e8970c7-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-m6qv8\" (UID: \"9de91381-a29d-422d-b859-69229e8970c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6qv8" Nov 01 20:06:22.918853 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:22.918824 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrhdv\" (UniqueName: \"kubernetes.io/projected/9de91381-a29d-422d-b859-69229e8970c7-kube-api-access-vrhdv\") pod \"maas-default-gateway-openshift-default-fd4b99879-m6qv8\" (UID: \"9de91381-a29d-422d-b859-69229e8970c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6qv8" Nov 01 20:06:22.919555 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:22.919534 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9de91381-a29d-422d-b859-69229e8970c7-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-m6qv8\" (UID: \"9de91381-a29d-422d-b859-69229e8970c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6qv8" Nov 01 20:06:23.041385 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:23.041312 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6qv8" Nov 01 20:06:23.315430 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:23.315405 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6qv8"] Nov 01 20:06:23.317863 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:06:23.317836 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9de91381_a29d_422d_b859_69229e8970c7.slice/crio-b1e85fd42e2d118c0cc697dc5289215e91aed94fac604ebaebcc6a89379bb165 WatchSource:0}: Error finding container b1e85fd42e2d118c0cc697dc5289215e91aed94fac604ebaebcc6a89379bb165: Status 404 returned error can't find the container with id b1e85fd42e2d118c0cc697dc5289215e91aed94fac604ebaebcc6a89379bb165 Nov 01 20:06:23.320034 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:23.320001 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:06:23.320107 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:23.320071 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:06:23.320107 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:23.320102 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:06:23.614069 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:23.613988 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6qv8" event={"ID":"9de91381-a29d-422d-b859-69229e8970c7","Type":"ContainerStarted","Data":"bba7828e82d937184a51d2b56823df558af30dec5d7d0810a5275b9c1a959965"} Nov 01 20:06:23.614069 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:23.614049 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6qv8" event={"ID":"9de91381-a29d-422d-b859-69229e8970c7","Type":"ContainerStarted","Data":"b1e85fd42e2d118c0cc697dc5289215e91aed94fac604ebaebcc6a89379bb165"} Nov 01 20:06:23.633858 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:23.633811 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6qv8" podStartSLOduration=1.633788967 podStartE2EDuration="1.633788967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:06:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:06:23.633092061 +0000 UTC m=+429.406252573" watchObservedRunningTime="2026-11-01 20:06:23.633788967 +0000 UTC m=+429.406949480" Nov 01 20:06:24.041687 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:24.041650 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6qv8" Nov 01 20:06:24.043238 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:24.043209 5935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-m6qv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 01 20:06:24.043477 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:24.043445 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6qv8" podUID="9de91381-a29d-422d-b859-69229e8970c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 01 20:06:25.042011 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:25.041979 5935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-m6qv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 01 20:06:25.042450 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:25.042033 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6qv8" podUID="9de91381-a29d-422d-b859-69229e8970c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 01 20:06:26.042652 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:26.042617 5935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-m6qv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 01 20:06:26.043215 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:26.043183 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6qv8" podUID="9de91381-a29d-422d-b859-69229e8970c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 01 20:06:27.042636 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:27.042600 5935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-m6qv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 01 20:06:27.042911 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:27.042880 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6qv8" podUID="9de91381-a29d-422d-b859-69229e8970c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 01 20:06:28.042071 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:28.042037 5935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-m6qv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 01 20:06:28.042249 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:28.042087 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6qv8" podUID="9de91381-a29d-422d-b859-69229e8970c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 01 20:06:29.041822 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:29.041779 5935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-m6qv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 01 20:06:29.042222 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:29.041855 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6qv8" podUID="9de91381-a29d-422d-b859-69229e8970c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 01 20:06:30.041784 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:30.041752 5935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-m6qv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Nov 01 20:06:30.041957 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:30.041826 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6qv8" podUID="9de91381-a29d-422d-b859-69229e8970c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Nov 01 20:06:31.556591 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:31.556560 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6qv8" Nov 01 20:06:31.557055 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:31.556946 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6qv8" Nov 01 20:06:31.557850 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:31.557830 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6qv8" Nov 01 20:06:31.716516 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:31.716486 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-e27ae7-7c877fb66c-hglrt"] Nov 01 20:06:31.719874 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:31.719857 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e27ae7-7c877fb66c-hglrt" Nov 01 20:06:31.733588 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:31.733562 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-e27ae7-dockercfg-f6jgt\"" Nov 01 20:06:31.734565 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:31.734544 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-e27ae7\"" Nov 01 20:06:31.737194 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:31.737169 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-e27ae7-7c877fb66c-hglrt"] Nov 01 20:06:31.809361 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:31.809258 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwlpw\" (UniqueName: \"kubernetes.io/projected/b7d68b3d-ba87-4c71-9399-989f83e198b4-kube-api-access-wwlpw\") pod \"payload-pre-processing-e2e-worker-w2-e27ae7-7c877fb66c-hglrt\" (UID: \"b7d68b3d-ba87-4c71-9399-989f83e198b4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e27ae7-7c877fb66c-hglrt" Nov 01 20:06:31.809361 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:31.809347 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7d68b3d-ba87-4c71-9399-989f83e198b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-e27ae7-7c877fb66c-hglrt\" (UID: \"b7d68b3d-ba87-4c71-9399-989f83e198b4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e27ae7-7c877fb66c-hglrt" Nov 01 20:06:31.910225 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:31.910192 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwlpw\" (UniqueName: \"kubernetes.io/projected/b7d68b3d-ba87-4c71-9399-989f83e198b4-kube-api-access-wwlpw\") pod \"payload-pre-processing-e2e-worker-w2-e27ae7-7c877fb66c-hglrt\" (UID: \"b7d68b3d-ba87-4c71-9399-989f83e198b4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e27ae7-7c877fb66c-hglrt" Nov 01 20:06:31.910425 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:31.910235 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7d68b3d-ba87-4c71-9399-989f83e198b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-e27ae7-7c877fb66c-hglrt\" (UID: \"b7d68b3d-ba87-4c71-9399-989f83e198b4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e27ae7-7c877fb66c-hglrt" Nov 01 20:06:31.910956 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:31.910935 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7d68b3d-ba87-4c71-9399-989f83e198b4-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-e27ae7-7c877fb66c-hglrt\" (UID: \"b7d68b3d-ba87-4c71-9399-989f83e198b4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e27ae7-7c877fb66c-hglrt" Nov 01 20:06:31.918100 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:31.918073 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwlpw\" (UniqueName: \"kubernetes.io/projected/b7d68b3d-ba87-4c71-9399-989f83e198b4-kube-api-access-wwlpw\") pod \"payload-pre-processing-e2e-worker-w2-e27ae7-7c877fb66c-hglrt\" (UID: \"b7d68b3d-ba87-4c71-9399-989f83e198b4\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e27ae7-7c877fb66c-hglrt" Nov 01 20:06:32.035355 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:32.035323 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e27ae7-7c877fb66c-hglrt" Nov 01 20:06:32.312068 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:32.312052 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-e27ae7-7c877fb66c-hglrt"] Nov 01 20:06:32.315812 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:06:32.315773 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7d68b3d_ba87_4c71_9399_989f83e198b4.slice/crio-c7df082b1e1d8ea5c9e9d9c8206dad35f65873faeb5db27c4d352fde9a929dbe WatchSource:0}: Error finding container c7df082b1e1d8ea5c9e9d9c8206dad35f65873faeb5db27c4d352fde9a929dbe: Status 404 returned error can't find the container with id c7df082b1e1d8ea5c9e9d9c8206dad35f65873faeb5db27c4d352fde9a929dbe Nov 01 20:06:32.317578 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:32.317560 5935 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 20:06:32.333083 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:32.333059 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-0aa61b-5c479c87bc-jsktt"] Nov 01 20:06:32.337627 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:32.337586 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0aa61b-5c479c87bc-jsktt" Nov 01 20:06:32.344341 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:32.344319 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-0aa61b-5c479c87bc-jsktt"] Nov 01 20:06:32.344722 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:32.344703 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-0aa61b\"" Nov 01 20:06:32.344919 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:32.344903 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-0aa61b-dockercfg-5grfn\"" Nov 01 20:06:32.414626 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:32.414590 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6vtn\" (UniqueName: \"kubernetes.io/projected/81191553-1912-45b2-a58a-01fc64d43e23-kube-api-access-t6vtn\") pod \"payload-pre-processing-e2e-worker-w4-0aa61b-5c479c87bc-jsktt\" (UID: \"81191553-1912-45b2-a58a-01fc64d43e23\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0aa61b-5c479c87bc-jsktt" Nov 01 20:06:32.414781 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:32.414762 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81191553-1912-45b2-a58a-01fc64d43e23-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-0aa61b-5c479c87bc-jsktt\" (UID: \"81191553-1912-45b2-a58a-01fc64d43e23\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0aa61b-5c479c87bc-jsktt" Nov 01 20:06:32.515304 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:32.515253 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81191553-1912-45b2-a58a-01fc64d43e23-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-0aa61b-5c479c87bc-jsktt\" (UID: \"81191553-1912-45b2-a58a-01fc64d43e23\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0aa61b-5c479c87bc-jsktt" Nov 01 20:06:32.515477 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:32.515336 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6vtn\" (UniqueName: \"kubernetes.io/projected/81191553-1912-45b2-a58a-01fc64d43e23-kube-api-access-t6vtn\") pod \"payload-pre-processing-e2e-worker-w4-0aa61b-5c479c87bc-jsktt\" (UID: \"81191553-1912-45b2-a58a-01fc64d43e23\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0aa61b-5c479c87bc-jsktt" Nov 01 20:06:32.515963 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:32.515942 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81191553-1912-45b2-a58a-01fc64d43e23-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-0aa61b-5c479c87bc-jsktt\" (UID: \"81191553-1912-45b2-a58a-01fc64d43e23\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0aa61b-5c479c87bc-jsktt" Nov 01 20:06:32.526048 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:32.526019 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6vtn\" (UniqueName: \"kubernetes.io/projected/81191553-1912-45b2-a58a-01fc64d43e23-kube-api-access-t6vtn\") pod \"payload-pre-processing-e2e-worker-w4-0aa61b-5c479c87bc-jsktt\" (UID: \"81191553-1912-45b2-a58a-01fc64d43e23\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0aa61b-5c479c87bc-jsktt" Nov 01 20:06:32.653345 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:32.653242 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0aa61b-5c479c87bc-jsktt" Nov 01 20:06:32.660832 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:32.660774 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e27ae7-7c877fb66c-hglrt" event={"ID":"b7d68b3d-ba87-4c71-9399-989f83e198b4","Type":"ContainerStarted","Data":"c7df082b1e1d8ea5c9e9d9c8206dad35f65873faeb5db27c4d352fde9a929dbe"} Nov 01 20:06:32.930812 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:32.930747 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-0aa61b-5c479c87bc-jsktt"] Nov 01 20:06:32.934965 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:06:32.934935 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81191553_1912_45b2_a58a_01fc64d43e23.slice/crio-b97f5b832c9290236dd6c97f1b22cd05a67e8bb7f374e9e5d9bb499b6900429f WatchSource:0}: Error finding container b97f5b832c9290236dd6c97f1b22cd05a67e8bb7f374e9e5d9bb499b6900429f: Status 404 returned error can't find the container with id b97f5b832c9290236dd6c97f1b22cd05a67e8bb7f374e9e5d9bb499b6900429f Nov 01 20:06:33.122184 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:33.122149 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-e27ae7-8595cd7889-ttqzl"] Nov 01 20:06:33.129535 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:33.129510 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e27ae7-8595cd7889-ttqzl" Nov 01 20:06:33.139205 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:33.139150 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 01 20:06:33.139205 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:33.139169 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7vx47\"" Nov 01 20:06:33.140054 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:33.139228 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-e27ae7-serving-cert\"" Nov 01 20:06:33.142279 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:33.140800 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-e27ae7-8595cd7889-ttqzl"] Nov 01 20:06:33.222507 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:33.222470 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftpzt\" (UniqueName: \"kubernetes.io/projected/77d9168b-ce7b-47ce-999c-9f9622472d48-kube-api-access-ftpzt\") pod \"maas-api-e2e-worker-w2-e27ae7-8595cd7889-ttqzl\" (UID: \"77d9168b-ce7b-47ce-999c-9f9622472d48\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e27ae7-8595cd7889-ttqzl" Nov 01 20:06:33.222683 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:33.222529 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77d9168b-ce7b-47ce-999c-9f9622472d48-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-e27ae7-8595cd7889-ttqzl\" (UID: \"77d9168b-ce7b-47ce-999c-9f9622472d48\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e27ae7-8595cd7889-ttqzl" Nov 01 20:06:33.222683 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:33.222641 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77d9168b-ce7b-47ce-999c-9f9622472d48-metrics-certs\") pod \"maas-api-e2e-worker-w2-e27ae7-8595cd7889-ttqzl\" (UID: \"77d9168b-ce7b-47ce-999c-9f9622472d48\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e27ae7-8595cd7889-ttqzl" Nov 01 20:06:33.222805 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:33.222769 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77d9168b-ce7b-47ce-999c-9f9622472d48-maas-api-tls\") pod \"maas-api-e2e-worker-w2-e27ae7-8595cd7889-ttqzl\" (UID: \"77d9168b-ce7b-47ce-999c-9f9622472d48\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e27ae7-8595cd7889-ttqzl" Nov 01 20:06:33.324886 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:33.324269 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77d9168b-ce7b-47ce-999c-9f9622472d48-maas-api-tls\") pod \"maas-api-e2e-worker-w2-e27ae7-8595cd7889-ttqzl\" (UID: \"77d9168b-ce7b-47ce-999c-9f9622472d48\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e27ae7-8595cd7889-ttqzl" Nov 01 20:06:33.324886 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:33.324367 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftpzt\" (UniqueName: \"kubernetes.io/projected/77d9168b-ce7b-47ce-999c-9f9622472d48-kube-api-access-ftpzt\") pod \"maas-api-e2e-worker-w2-e27ae7-8595cd7889-ttqzl\" (UID: \"77d9168b-ce7b-47ce-999c-9f9622472d48\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e27ae7-8595cd7889-ttqzl" Nov 01 20:06:33.324886 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:33.324421 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77d9168b-ce7b-47ce-999c-9f9622472d48-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-e27ae7-8595cd7889-ttqzl\" (UID: \"77d9168b-ce7b-47ce-999c-9f9622472d48\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e27ae7-8595cd7889-ttqzl" Nov 01 20:06:33.324886 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:33.324452 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77d9168b-ce7b-47ce-999c-9f9622472d48-metrics-certs\") pod \"maas-api-e2e-worker-w2-e27ae7-8595cd7889-ttqzl\" (UID: \"77d9168b-ce7b-47ce-999c-9f9622472d48\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e27ae7-8595cd7889-ttqzl" Nov 01 20:06:33.326004 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:33.325950 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77d9168b-ce7b-47ce-999c-9f9622472d48-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-e27ae7-8595cd7889-ttqzl\" (UID: \"77d9168b-ce7b-47ce-999c-9f9622472d48\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e27ae7-8595cd7889-ttqzl" Nov 01 20:06:33.327811 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:33.327786 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77d9168b-ce7b-47ce-999c-9f9622472d48-maas-api-tls\") pod \"maas-api-e2e-worker-w2-e27ae7-8595cd7889-ttqzl\" (UID: \"77d9168b-ce7b-47ce-999c-9f9622472d48\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e27ae7-8595cd7889-ttqzl" Nov 01 20:06:33.334906 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:33.334882 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftpzt\" (UniqueName: \"kubernetes.io/projected/77d9168b-ce7b-47ce-999c-9f9622472d48-kube-api-access-ftpzt\") pod \"maas-api-e2e-worker-w2-e27ae7-8595cd7889-ttqzl\" (UID: \"77d9168b-ce7b-47ce-999c-9f9622472d48\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e27ae7-8595cd7889-ttqzl" Nov 01 20:06:33.337221 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:33.337196 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77d9168b-ce7b-47ce-999c-9f9622472d48-metrics-certs\") pod \"maas-api-e2e-worker-w2-e27ae7-8595cd7889-ttqzl\" (UID: \"77d9168b-ce7b-47ce-999c-9f9622472d48\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e27ae7-8595cd7889-ttqzl" Nov 01 20:06:33.452616 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:33.452586 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e27ae7-8595cd7889-ttqzl" Nov 01 20:06:33.668381 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:33.668346 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0aa61b-5c479c87bc-jsktt" event={"ID":"81191553-1912-45b2-a58a-01fc64d43e23","Type":"ContainerStarted","Data":"b97f5b832c9290236dd6c97f1b22cd05a67e8bb7f374e9e5d9bb499b6900429f"} Nov 01 20:06:33.745000 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:33.744972 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-e27ae7-8595cd7889-ttqzl"] Nov 01 20:06:33.748366 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:06:33.748337 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77d9168b_ce7b_47ce_999c_9f9622472d48.slice/crio-56dd5b98dd721405c00f63c304332f060f12d8cbb249be04e18ffba66e836d0f WatchSource:0}: Error finding container 56dd5b98dd721405c00f63c304332f060f12d8cbb249be04e18ffba66e836d0f: Status 404 returned error can't find the container with id 56dd5b98dd721405c00f63c304332f060f12d8cbb249be04e18ffba66e836d0f Nov 01 20:06:33.816964 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:33.816655 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-0aa61b-74c6dc5659-g7srp"] Nov 01 20:06:33.822598 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:33.822569 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0aa61b-74c6dc5659-g7srp" Nov 01 20:06:33.829252 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:33.829226 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-0aa61b-74c6dc5659-g7srp"] Nov 01 20:06:33.832596 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:33.832557 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-0aa61b-serving-cert\"" Nov 01 20:06:33.934364 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:33.934334 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs4lq\" (UniqueName: \"kubernetes.io/projected/d8569d8c-c04e-47dd-895c-2359353049ae-kube-api-access-bs4lq\") pod \"maas-api-e2e-worker-w4-0aa61b-74c6dc5659-g7srp\" (UID: \"d8569d8c-c04e-47dd-895c-2359353049ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0aa61b-74c6dc5659-g7srp" Nov 01 20:06:33.934557 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:33.934399 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8569d8c-c04e-47dd-895c-2359353049ae-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-0aa61b-74c6dc5659-g7srp\" (UID: \"d8569d8c-c04e-47dd-895c-2359353049ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0aa61b-74c6dc5659-g7srp" Nov 01 20:06:33.934557 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:33.934541 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8569d8c-c04e-47dd-895c-2359353049ae-maas-api-tls\") pod \"maas-api-e2e-worker-w4-0aa61b-74c6dc5659-g7srp\" (UID: \"d8569d8c-c04e-47dd-895c-2359353049ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0aa61b-74c6dc5659-g7srp" Nov 01 20:06:33.934665 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:33.934580 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8569d8c-c04e-47dd-895c-2359353049ae-metrics-certs\") pod \"maas-api-e2e-worker-w4-0aa61b-74c6dc5659-g7srp\" (UID: \"d8569d8c-c04e-47dd-895c-2359353049ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0aa61b-74c6dc5659-g7srp" Nov 01 20:06:34.035313 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:34.035246 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8569d8c-c04e-47dd-895c-2359353049ae-maas-api-tls\") pod \"maas-api-e2e-worker-w4-0aa61b-74c6dc5659-g7srp\" (UID: \"d8569d8c-c04e-47dd-895c-2359353049ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0aa61b-74c6dc5659-g7srp" Nov 01 20:06:34.035561 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:34.035319 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8569d8c-c04e-47dd-895c-2359353049ae-metrics-certs\") pod \"maas-api-e2e-worker-w4-0aa61b-74c6dc5659-g7srp\" (UID: \"d8569d8c-c04e-47dd-895c-2359353049ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0aa61b-74c6dc5659-g7srp" Nov 01 20:06:34.035561 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:34.035446 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs4lq\" (UniqueName: \"kubernetes.io/projected/d8569d8c-c04e-47dd-895c-2359353049ae-kube-api-access-bs4lq\") pod \"maas-api-e2e-worker-w4-0aa61b-74c6dc5659-g7srp\" (UID: \"d8569d8c-c04e-47dd-895c-2359353049ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0aa61b-74c6dc5659-g7srp" Nov 01 20:06:34.035561 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:34.035498 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8569d8c-c04e-47dd-895c-2359353049ae-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-0aa61b-74c6dc5659-g7srp\" (UID: \"d8569d8c-c04e-47dd-895c-2359353049ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0aa61b-74c6dc5659-g7srp" Nov 01 20:06:34.036306 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:34.036254 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8569d8c-c04e-47dd-895c-2359353049ae-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-0aa61b-74c6dc5659-g7srp\" (UID: \"d8569d8c-c04e-47dd-895c-2359353049ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0aa61b-74c6dc5659-g7srp" Nov 01 20:06:34.039001 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:34.038977 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8569d8c-c04e-47dd-895c-2359353049ae-maas-api-tls\") pod \"maas-api-e2e-worker-w4-0aa61b-74c6dc5659-g7srp\" (UID: \"d8569d8c-c04e-47dd-895c-2359353049ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0aa61b-74c6dc5659-g7srp" Nov 01 20:06:34.039469 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:34.039442 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8569d8c-c04e-47dd-895c-2359353049ae-metrics-certs\") pod \"maas-api-e2e-worker-w4-0aa61b-74c6dc5659-g7srp\" (UID: \"d8569d8c-c04e-47dd-895c-2359353049ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0aa61b-74c6dc5659-g7srp" Nov 01 20:06:34.046419 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:34.046395 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs4lq\" (UniqueName: \"kubernetes.io/projected/d8569d8c-c04e-47dd-895c-2359353049ae-kube-api-access-bs4lq\") pod \"maas-api-e2e-worker-w4-0aa61b-74c6dc5659-g7srp\" (UID: \"d8569d8c-c04e-47dd-895c-2359353049ae\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0aa61b-74c6dc5659-g7srp" Nov 01 20:06:34.151546 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:34.151467 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0aa61b-74c6dc5659-g7srp" Nov 01 20:06:34.457569 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:34.456888 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-0aa61b-74c6dc5659-g7srp"] Nov 01 20:06:34.460063 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:06:34.460035 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8569d8c_c04e_47dd_895c_2359353049ae.slice/crio-86d7d0ea4ba41dcab25723000764a769052efa7b016fd02e00d2900f775435dd WatchSource:0}: Error finding container 86d7d0ea4ba41dcab25723000764a769052efa7b016fd02e00d2900f775435dd: Status 404 returned error can't find the container with id 86d7d0ea4ba41dcab25723000764a769052efa7b016fd02e00d2900f775435dd Nov 01 20:06:34.684394 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:34.684342 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0aa61b-74c6dc5659-g7srp" event={"ID":"d8569d8c-c04e-47dd-895c-2359353049ae","Type":"ContainerStarted","Data":"86d7d0ea4ba41dcab25723000764a769052efa7b016fd02e00d2900f775435dd"} Nov 01 20:06:34.686965 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:34.686884 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e27ae7-8595cd7889-ttqzl" event={"ID":"77d9168b-ce7b-47ce-999c-9f9622472d48","Type":"ContainerStarted","Data":"d7cd6a447ac638d560f77bf4981c73ac785af399b2acd11214872c4669521d9a"} Nov 01 20:06:34.686965 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:34.686935 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e27ae7-8595cd7889-ttqzl" event={"ID":"77d9168b-ce7b-47ce-999c-9f9622472d48","Type":"ContainerStarted","Data":"56dd5b98dd721405c00f63c304332f060f12d8cbb249be04e18ffba66e836d0f"} Nov 01 20:06:34.687180 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:34.687162 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e27ae7-8595cd7889-ttqzl" Nov 01 20:06:34.704595 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:34.704539 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e27ae7-8595cd7889-ttqzl" podStartSLOduration=1.142040534 podStartE2EDuration="1.704520734s" totalImagesPullingTime="562.4802ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:06:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:06:34.703665665 +0000 UTC m=+440.476826179" watchObservedRunningTime="2026-11-01 20:06:34.704520734 +0000 UTC m=+440.477681243" Nov 01 20:06:35.693647 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:35.693613 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0aa61b-74c6dc5659-g7srp" event={"ID":"d8569d8c-c04e-47dd-895c-2359353049ae","Type":"ContainerStarted","Data":"27b809f376135693042692804138141852d0aa1be0995d93b2ca06e0d25bbc61"} Nov 01 20:06:35.694062 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:35.693673 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0aa61b-74c6dc5659-g7srp" Nov 01 20:06:35.695191 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:35.695166 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e27ae7-7c877fb66c-hglrt" event={"ID":"b7d68b3d-ba87-4c71-9399-989f83e198b4","Type":"ContainerStarted","Data":"a8d6eee555e6d0a20c087b0b50848246eb6b8dcb0b4dc2f752041511e25246fc"} Nov 01 20:06:35.695323 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:35.695249 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e27ae7-7c877fb66c-hglrt" Nov 01 20:06:35.696686 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:35.696664 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0aa61b-5c479c87bc-jsktt" event={"ID":"81191553-1912-45b2-a58a-01fc64d43e23","Type":"ContainerStarted","Data":"3fcac3ac815678d5b9652691475484c922487cc98ced7fa93d6dc3c8ef3b7e4e"} Nov 01 20:06:35.696853 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:35.696839 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0aa61b-5c479c87bc-jsktt" Nov 01 20:06:35.716380 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:35.716342 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0aa61b-74c6dc5659-g7srp" podStartSLOduration=2.263964423 podStartE2EDuration="2.716330381s" totalImagesPullingTime="452.365958ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:06:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:06:35.715948319 +0000 UTC m=+441.489108832" watchObservedRunningTime="2026-11-01 20:06:35.716330381 +0000 UTC m=+441.489490892" Nov 01 20:06:35.731422 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:35.731387 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e27ae7-7c877fb66c-hglrt" podStartSLOduration=2.128830202 podStartE2EDuration="4.731380136s" totalImagesPullingTime="2.602549934s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:06:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:06:35.730097692 +0000 UTC m=+441.503258205" watchObservedRunningTime="2026-11-01 20:06:35.731380136 +0000 UTC m=+441.504540647" Nov 01 20:06:35.746919 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:35.746882 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0aa61b-5c479c87bc-jsktt" podStartSLOduration=1.412560107 podStartE2EDuration="3.746872378s" totalImagesPullingTime="2.334312271s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:06:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:06:35.744586732 +0000 UTC m=+441.517747244" watchObservedRunningTime="2026-11-01 20:06:35.746872378 +0000 UTC m=+441.520032890" Nov 01 20:06:35.843132 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:06:35.843101 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:06:37.737427 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:37.737396 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-fl2rn"] Nov 01 20:06:37.741141 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:37.741113 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-fl2rn" Nov 01 20:06:37.753569 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:37.753542 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-fl2rn"] Nov 01 20:06:37.874128 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:37.874093 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/043fcc04-22eb-454f-a1d5-d669214c5b0c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-fl2rn\" (UID: \"043fcc04-22eb-454f-a1d5-d669214c5b0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-fl2rn" Nov 01 20:06:37.874128 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:37.874132 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/043fcc04-22eb-454f-a1d5-d669214c5b0c-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-fl2rn\" (UID: \"043fcc04-22eb-454f-a1d5-d669214c5b0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-fl2rn" Nov 01 20:06:37.874360 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:37.874227 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlw4b\" (UniqueName: \"kubernetes.io/projected/043fcc04-22eb-454f-a1d5-d669214c5b0c-kube-api-access-hlw4b\") pod \"maas-default-gateway-openshift-default-fd4b99879-fl2rn\" (UID: \"043fcc04-22eb-454f-a1d5-d669214c5b0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-fl2rn" Nov 01 20:06:37.874360 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:37.874268 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/043fcc04-22eb-454f-a1d5-d669214c5b0c-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-fl2rn\" (UID: \"043fcc04-22eb-454f-a1d5-d669214c5b0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-fl2rn" Nov 01 20:06:37.874435 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:37.874353 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/043fcc04-22eb-454f-a1d5-d669214c5b0c-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-fl2rn\" (UID: \"043fcc04-22eb-454f-a1d5-d669214c5b0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-fl2rn" Nov 01 20:06:37.874435 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:37.874398 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/043fcc04-22eb-454f-a1d5-d669214c5b0c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-fl2rn\" (UID: \"043fcc04-22eb-454f-a1d5-d669214c5b0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-fl2rn" Nov 01 20:06:37.874497 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:37.874433 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/043fcc04-22eb-454f-a1d5-d669214c5b0c-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-fl2rn\" (UID: \"043fcc04-22eb-454f-a1d5-d669214c5b0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-fl2rn" Nov 01 20:06:37.874529 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:37.874514 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/043fcc04-22eb-454f-a1d5-d669214c5b0c-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-fl2rn\" (UID: \"043fcc04-22eb-454f-a1d5-d669214c5b0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-fl2rn" Nov 01 20:06:37.874568 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:37.874544 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/043fcc04-22eb-454f-a1d5-d669214c5b0c-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-fl2rn\" (UID: \"043fcc04-22eb-454f-a1d5-d669214c5b0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-fl2rn" Nov 01 20:06:37.975303 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:37.975254 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/043fcc04-22eb-454f-a1d5-d669214c5b0c-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-fl2rn\" (UID: \"043fcc04-22eb-454f-a1d5-d669214c5b0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-fl2rn" Nov 01 20:06:37.975453 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:37.975335 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/043fcc04-22eb-454f-a1d5-d669214c5b0c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-fl2rn\" (UID: \"043fcc04-22eb-454f-a1d5-d669214c5b0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-fl2rn" Nov 01 20:06:37.975453 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:37.975370 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/043fcc04-22eb-454f-a1d5-d669214c5b0c-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-fl2rn\" (UID: \"043fcc04-22eb-454f-a1d5-d669214c5b0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-fl2rn" Nov 01 20:06:37.975453 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:37.975404 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/043fcc04-22eb-454f-a1d5-d669214c5b0c-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-fl2rn\" (UID: \"043fcc04-22eb-454f-a1d5-d669214c5b0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-fl2rn" Nov 01 20:06:37.975453 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:37.975441 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/043fcc04-22eb-454f-a1d5-d669214c5b0c-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-fl2rn\" (UID: \"043fcc04-22eb-454f-a1d5-d669214c5b0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-fl2rn" Nov 01 20:06:37.975728 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:37.975471 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/043fcc04-22eb-454f-a1d5-d669214c5b0c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-fl2rn\" (UID: \"043fcc04-22eb-454f-a1d5-d669214c5b0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-fl2rn" Nov 01 20:06:37.975728 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:37.975607 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/043fcc04-22eb-454f-a1d5-d669214c5b0c-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-fl2rn\" (UID: \"043fcc04-22eb-454f-a1d5-d669214c5b0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-fl2rn" Nov 01 20:06:37.975728 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:37.975698 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlw4b\" (UniqueName: \"kubernetes.io/projected/043fcc04-22eb-454f-a1d5-d669214c5b0c-kube-api-access-hlw4b\") pod \"maas-default-gateway-openshift-default-fd4b99879-fl2rn\" (UID: \"043fcc04-22eb-454f-a1d5-d669214c5b0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-fl2rn" Nov 01 20:06:37.975862 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:37.975737 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/043fcc04-22eb-454f-a1d5-d669214c5b0c-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-fl2rn\" (UID: \"043fcc04-22eb-454f-a1d5-d669214c5b0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-fl2rn" Nov 01 20:06:37.975862 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:37.975752 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/043fcc04-22eb-454f-a1d5-d669214c5b0c-credential-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-fl2rn\" (UID: \"043fcc04-22eb-454f-a1d5-d669214c5b0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-fl2rn" Nov 01 20:06:37.975862 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:37.975784 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/043fcc04-22eb-454f-a1d5-d669214c5b0c-istio-data\") pod \"maas-default-gateway-openshift-default-fd4b99879-fl2rn\" (UID: \"043fcc04-22eb-454f-a1d5-d669214c5b0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-fl2rn" Nov 01 20:06:37.975988 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:37.975879 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/043fcc04-22eb-454f-a1d5-d669214c5b0c-workload-certs\") pod \"maas-default-gateway-openshift-default-fd4b99879-fl2rn\" (UID: \"043fcc04-22eb-454f-a1d5-d669214c5b0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-fl2rn" Nov 01 20:06:37.975988 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:37.975921 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/043fcc04-22eb-454f-a1d5-d669214c5b0c-workload-socket\") pod \"maas-default-gateway-openshift-default-fd4b99879-fl2rn\" (UID: \"043fcc04-22eb-454f-a1d5-d669214c5b0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-fl2rn" Nov 01 20:06:37.976128 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:37.976112 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/043fcc04-22eb-454f-a1d5-d669214c5b0c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-fd4b99879-fl2rn\" (UID: \"043fcc04-22eb-454f-a1d5-d669214c5b0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-fl2rn" Nov 01 20:06:37.977751 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:37.977730 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/043fcc04-22eb-454f-a1d5-d669214c5b0c-istio-envoy\") pod \"maas-default-gateway-openshift-default-fd4b99879-fl2rn\" (UID: \"043fcc04-22eb-454f-a1d5-d669214c5b0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-fl2rn" Nov 01 20:06:37.978052 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:37.978032 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/043fcc04-22eb-454f-a1d5-d669214c5b0c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-fd4b99879-fl2rn\" (UID: \"043fcc04-22eb-454f-a1d5-d669214c5b0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-fl2rn" Nov 01 20:06:37.983405 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:37.983381 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlw4b\" (UniqueName: \"kubernetes.io/projected/043fcc04-22eb-454f-a1d5-d669214c5b0c-kube-api-access-hlw4b\") pod \"maas-default-gateway-openshift-default-fd4b99879-fl2rn\" (UID: \"043fcc04-22eb-454f-a1d5-d669214c5b0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-fl2rn" Nov 01 20:06:37.983405 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:37.983394 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/043fcc04-22eb-454f-a1d5-d669214c5b0c-istio-token\") pod \"maas-default-gateway-openshift-default-fd4b99879-fl2rn\" (UID: \"043fcc04-22eb-454f-a1d5-d669214c5b0c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-fl2rn" Nov 01 20:06:38.055695 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:38.055625 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-fl2rn" Nov 01 20:06:38.332619 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:38.332535 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-fl2rn"] Nov 01 20:06:38.339708 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:06:38.339670 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod043fcc04_22eb_454f_a1d5_d669214c5b0c.slice/crio-c0b677663d05df96ba6bf281c2fc75d8e702bf75c3f8808133a5cb02929bd889 WatchSource:0}: Error finding container c0b677663d05df96ba6bf281c2fc75d8e702bf75c3f8808133a5cb02929bd889: Status 404 returned error can't find the container with id c0b677663d05df96ba6bf281c2fc75d8e702bf75c3f8808133a5cb02929bd889 Nov 01 20:06:38.342238 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:38.342198 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:06:38.342331 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:38.342303 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:06:38.342391 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:38.342372 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:06:38.715188 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:38.715150 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-fl2rn" event={"ID":"043fcc04-22eb-454f-a1d5-d669214c5b0c","Type":"ContainerStarted","Data":"a3f100a6588bd0bc83581bb584145a021527913f9cf449d9b1011bdd9891e49a"} Nov 01 20:06:38.715385 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:38.715214 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-fl2rn" event={"ID":"043fcc04-22eb-454f-a1d5-d669214c5b0c","Type":"ContainerStarted","Data":"c0b677663d05df96ba6bf281c2fc75d8e702bf75c3f8808133a5cb02929bd889"} Nov 01 20:06:38.745280 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:38.745122 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-fl2rn" podStartSLOduration=1.745098768 podStartE2EDuration="1.745098768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:06:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:06:38.740693108 +0000 UTC m=+444.513853619" watchObservedRunningTime="2026-11-01 20:06:38.745098768 +0000 UTC m=+444.518259281" Nov 01 20:06:39.067375 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:39.065897 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-fl2rn" Nov 01 20:06:39.068898 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:39.068866 5935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-fl2rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 01 20:06:39.069024 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:39.068919 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-fl2rn" podUID="043fcc04-22eb-454f-a1d5-d669214c5b0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 01 20:06:40.060681 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:40.060643 5935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-fl2rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 01 20:06:40.061256 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:40.061224 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-fl2rn" podUID="043fcc04-22eb-454f-a1d5-d669214c5b0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 01 20:06:40.702237 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:40.702210 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e27ae7-8595cd7889-ttqzl" Nov 01 20:06:41.060783 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:41.060690 5935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-fl2rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 01 20:06:41.061381 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:41.061347 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-fl2rn" podUID="043fcc04-22eb-454f-a1d5-d669214c5b0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 01 20:06:41.707771 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:41.707744 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0aa61b-74c6dc5659-g7srp" Nov 01 20:06:42.057841 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:42.057741 5935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-fl2rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 01 20:06:42.058096 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:42.058061 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-fl2rn" podUID="043fcc04-22eb-454f-a1d5-d669214c5b0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 01 20:06:43.060146 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:43.059677 5935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-fl2rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 01 20:06:43.060146 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:43.059746 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-fl2rn" podUID="043fcc04-22eb-454f-a1d5-d669214c5b0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 01 20:06:44.066696 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:44.066658 5935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-fl2rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 01 20:06:44.067340 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:44.067268 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-fl2rn" podUID="043fcc04-22eb-454f-a1d5-d669214c5b0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 01 20:06:45.056596 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:45.056557 5935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-fl2rn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Nov 01 20:06:45.056810 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:45.056613 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-fl2rn" podUID="043fcc04-22eb-454f-a1d5-d669214c5b0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Nov 01 20:06:46.704819 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:46.704792 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0aa61b-5c479c87bc-jsktt" Nov 01 20:06:46.705398 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:46.704864 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e27ae7-7c877fb66c-hglrt" Nov 01 20:06:46.766308 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:46.766257 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-fl2rn" Nov 01 20:06:46.766663 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:46.766640 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-fl2rn" Nov 01 20:06:46.767521 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:46.767503 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-fl2rn" Nov 01 20:06:52.531657 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:52.531623 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-0aa61b/simulator-w4-0aa61b-kserve-6774594d4b-qmjz8"] Nov 01 20:06:52.535630 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:52.535612 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0aa61b/simulator-w4-0aa61b-kserve-6774594d4b-qmjz8" Nov 01 20:06:52.541864 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:52.541835 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09414dc0-7742-48fa-a4b5-571199d99e15-dshm\") pod \"simulator-w4-0aa61b-kserve-6774594d4b-qmjz8\" (UID: \"09414dc0-7742-48fa-a4b5-571199d99e15\") " pod="e2e-models-e2e-worker-w4-0aa61b/simulator-w4-0aa61b-kserve-6774594d4b-qmjz8" Nov 01 20:06:52.542009 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:52.541887 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09414dc0-7742-48fa-a4b5-571199d99e15-tls-certs\") pod \"simulator-w4-0aa61b-kserve-6774594d4b-qmjz8\" (UID: \"09414dc0-7742-48fa-a4b5-571199d99e15\") " pod="e2e-models-e2e-worker-w4-0aa61b/simulator-w4-0aa61b-kserve-6774594d4b-qmjz8" Nov 01 20:06:52.542075 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:52.542038 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcvjm\" (UniqueName: \"kubernetes.io/projected/09414dc0-7742-48fa-a4b5-571199d99e15-kube-api-access-jcvjm\") pod \"simulator-w4-0aa61b-kserve-6774594d4b-qmjz8\" (UID: \"09414dc0-7742-48fa-a4b5-571199d99e15\") " pod="e2e-models-e2e-worker-w4-0aa61b/simulator-w4-0aa61b-kserve-6774594d4b-qmjz8" Nov 01 20:06:52.542139 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:52.542077 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09414dc0-7742-48fa-a4b5-571199d99e15-model-cache\") pod \"simulator-w4-0aa61b-kserve-6774594d4b-qmjz8\" (UID: \"09414dc0-7742-48fa-a4b5-571199d99e15\") " pod="e2e-models-e2e-worker-w4-0aa61b/simulator-w4-0aa61b-kserve-6774594d4b-qmjz8" Nov 01 20:06:52.542192 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:52.542136 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09414dc0-7742-48fa-a4b5-571199d99e15-home\") pod \"simulator-w4-0aa61b-kserve-6774594d4b-qmjz8\" (UID: \"09414dc0-7742-48fa-a4b5-571199d99e15\") " pod="e2e-models-e2e-worker-w4-0aa61b/simulator-w4-0aa61b-kserve-6774594d4b-qmjz8" Nov 01 20:06:52.542192 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:52.542159 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09414dc0-7742-48fa-a4b5-571199d99e15-tmp-dir\") pod \"simulator-w4-0aa61b-kserve-6774594d4b-qmjz8\" (UID: \"09414dc0-7742-48fa-a4b5-571199d99e15\") " pod="e2e-models-e2e-worker-w4-0aa61b/simulator-w4-0aa61b-kserve-6774594d4b-qmjz8" Nov 01 20:06:52.543881 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:52.543858 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-0aa61b\"/\"kube-root-ca.crt\"" Nov 01 20:06:52.544021 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:52.543915 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-0aa61b\"/\"openshift-service-ca.crt\"" Nov 01 20:06:52.547146 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:52.544659 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-0aa61b\"/\"default-dockercfg-dnkln\"" Nov 01 20:06:52.547146 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:52.544762 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-0aa61b\"/\"simulator-w4-0aa61b-kserve-self-signed-certs\"" Nov 01 20:06:52.547146 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:52.546515 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-0aa61b/simulator-w4-0aa61b-kserve-6774594d4b-qmjz8"] Nov 01 20:06:52.643216 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:52.643174 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09414dc0-7742-48fa-a4b5-571199d99e15-model-cache\") pod \"simulator-w4-0aa61b-kserve-6774594d4b-qmjz8\" (UID: \"09414dc0-7742-48fa-a4b5-571199d99e15\") " pod="e2e-models-e2e-worker-w4-0aa61b/simulator-w4-0aa61b-kserve-6774594d4b-qmjz8" Nov 01 20:06:52.643216 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:52.643218 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09414dc0-7742-48fa-a4b5-571199d99e15-home\") pod \"simulator-w4-0aa61b-kserve-6774594d4b-qmjz8\" (UID: \"09414dc0-7742-48fa-a4b5-571199d99e15\") " pod="e2e-models-e2e-worker-w4-0aa61b/simulator-w4-0aa61b-kserve-6774594d4b-qmjz8" Nov 01 20:06:52.643479 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:52.643239 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09414dc0-7742-48fa-a4b5-571199d99e15-tmp-dir\") pod \"simulator-w4-0aa61b-kserve-6774594d4b-qmjz8\" (UID: \"09414dc0-7742-48fa-a4b5-571199d99e15\") " pod="e2e-models-e2e-worker-w4-0aa61b/simulator-w4-0aa61b-kserve-6774594d4b-qmjz8" Nov 01 20:06:52.643479 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:52.643281 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09414dc0-7742-48fa-a4b5-571199d99e15-dshm\") pod \"simulator-w4-0aa61b-kserve-6774594d4b-qmjz8\" (UID: \"09414dc0-7742-48fa-a4b5-571199d99e15\") " pod="e2e-models-e2e-worker-w4-0aa61b/simulator-w4-0aa61b-kserve-6774594d4b-qmjz8" Nov 01 20:06:52.643479 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:52.643333 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09414dc0-7742-48fa-a4b5-571199d99e15-tls-certs\") pod \"simulator-w4-0aa61b-kserve-6774594d4b-qmjz8\" (UID: \"09414dc0-7742-48fa-a4b5-571199d99e15\") " pod="e2e-models-e2e-worker-w4-0aa61b/simulator-w4-0aa61b-kserve-6774594d4b-qmjz8" Nov 01 20:06:52.643479 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:52.643427 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcvjm\" (UniqueName: \"kubernetes.io/projected/09414dc0-7742-48fa-a4b5-571199d99e15-kube-api-access-jcvjm\") pod \"simulator-w4-0aa61b-kserve-6774594d4b-qmjz8\" (UID: \"09414dc0-7742-48fa-a4b5-571199d99e15\") " pod="e2e-models-e2e-worker-w4-0aa61b/simulator-w4-0aa61b-kserve-6774594d4b-qmjz8" Nov 01 20:06:52.643731 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:52.643703 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09414dc0-7742-48fa-a4b5-571199d99e15-home\") pod \"simulator-w4-0aa61b-kserve-6774594d4b-qmjz8\" (UID: \"09414dc0-7742-48fa-a4b5-571199d99e15\") " pod="e2e-models-e2e-worker-w4-0aa61b/simulator-w4-0aa61b-kserve-6774594d4b-qmjz8" Nov 01 20:06:52.643862 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:52.643744 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09414dc0-7742-48fa-a4b5-571199d99e15-tmp-dir\") pod \"simulator-w4-0aa61b-kserve-6774594d4b-qmjz8\" (UID: \"09414dc0-7742-48fa-a4b5-571199d99e15\") " pod="e2e-models-e2e-worker-w4-0aa61b/simulator-w4-0aa61b-kserve-6774594d4b-qmjz8" Nov 01 20:06:52.644076 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:52.644051 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09414dc0-7742-48fa-a4b5-571199d99e15-model-cache\") pod \"simulator-w4-0aa61b-kserve-6774594d4b-qmjz8\" (UID: \"09414dc0-7742-48fa-a4b5-571199d99e15\") " pod="e2e-models-e2e-worker-w4-0aa61b/simulator-w4-0aa61b-kserve-6774594d4b-qmjz8" Nov 01 20:06:52.645691 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:52.645663 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09414dc0-7742-48fa-a4b5-571199d99e15-dshm\") pod \"simulator-w4-0aa61b-kserve-6774594d4b-qmjz8\" (UID: \"09414dc0-7742-48fa-a4b5-571199d99e15\") " pod="e2e-models-e2e-worker-w4-0aa61b/simulator-w4-0aa61b-kserve-6774594d4b-qmjz8" Nov 01 20:06:52.645945 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:52.645927 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09414dc0-7742-48fa-a4b5-571199d99e15-tls-certs\") pod \"simulator-w4-0aa61b-kserve-6774594d4b-qmjz8\" (UID: \"09414dc0-7742-48fa-a4b5-571199d99e15\") " pod="e2e-models-e2e-worker-w4-0aa61b/simulator-w4-0aa61b-kserve-6774594d4b-qmjz8" Nov 01 20:06:52.652477 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:52.652450 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcvjm\" (UniqueName: \"kubernetes.io/projected/09414dc0-7742-48fa-a4b5-571199d99e15-kube-api-access-jcvjm\") pod \"simulator-w4-0aa61b-kserve-6774594d4b-qmjz8\" (UID: \"09414dc0-7742-48fa-a4b5-571199d99e15\") " pod="e2e-models-e2e-worker-w4-0aa61b/simulator-w4-0aa61b-kserve-6774594d4b-qmjz8" Nov 01 20:06:52.854128 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:52.854050 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0aa61b/simulator-w4-0aa61b-kserve-6774594d4b-qmjz8" Nov 01 20:06:53.130665 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:53.130387 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-0aa61b/simulator-w4-0aa61b-kserve-6774594d4b-qmjz8"] Nov 01 20:06:53.133898 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:06:53.133866 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod09414dc0_7742_48fa_a4b5_571199d99e15.slice/crio-1de533698ac5b0d09f1c8258d961c93943893dba8e54f5f7bf0b524a65b4780a WatchSource:0}: Error finding container 1de533698ac5b0d09f1c8258d961c93943893dba8e54f5f7bf0b524a65b4780a: Status 404 returned error can't find the container with id 1de533698ac5b0d09f1c8258d961c93943893dba8e54f5f7bf0b524a65b4780a Nov 01 20:06:53.826185 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:53.826147 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0aa61b/simulator-w4-0aa61b-kserve-6774594d4b-qmjz8" event={"ID":"09414dc0-7742-48fa-a4b5-571199d99e15","Type":"ContainerStarted","Data":"1de533698ac5b0d09f1c8258d961c93943893dba8e54f5f7bf0b524a65b4780a"} Nov 01 20:06:54.833671 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:54.833578 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0aa61b/simulator-w4-0aa61b-kserve-6774594d4b-qmjz8" event={"ID":"09414dc0-7742-48fa-a4b5-571199d99e15","Type":"ContainerStarted","Data":"d46de431b842bd003775071eb7b0a1d4cae6f34915597b435d7718ac404aac13"} Nov 01 20:06:54.853330 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:54.853271 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-0aa61b/simulator-w4-0aa61b-kserve-6774594d4b-qmjz8" podStartSLOduration=1.647741023 podStartE2EDuration="2.853257853s" totalImagesPullingTime="1.20551683s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:06:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:06:54.851091304 +0000 UTC m=+460.624251820" watchObservedRunningTime="2026-11-01 20:06:54.853257853 +0000 UTC m=+460.626418364" Nov 01 20:06:54.854350 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:06:54.854334 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0aa61b/simulator-w4-0aa61b-kserve-6774594d4b-qmjz8" Nov 01 20:07:02.855181 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:02.855140 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-0aa61b/simulator-w4-0aa61b-kserve-6774594d4b-qmjz8" Nov 01 20:07:02.867984 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:02.867956 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-0aa61b/simulator-w4-0aa61b-kserve-6774594d4b-qmjz8" Nov 01 20:07:02.888486 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:02.888462 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0aa61b/simulator-w4-0aa61b-kserve-6774594d4b-qmjz8" Nov 01 20:07:05.843927 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:07:05.843887 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:07:17.141571 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:17.141538 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e3c4a7-57c4dc9874-swj7w"] Nov 01 20:07:17.149830 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:17.149798 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e3c4a7-57c4dc9874-swj7w" Nov 01 20:07:17.155646 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:17.155619 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e3c4a7-57c4dc9874-swj7w"] Nov 01 20:07:17.155816 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:17.155800 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-e3c4a7-serving-cert\"" Nov 01 20:07:17.295121 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:17.295088 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtbdw\" (UniqueName: \"kubernetes.io/projected/2f7b4b6e-9a7c-4ce8-aa0b-600cded35eb3-kube-api-access-wtbdw\") pod \"maas-api-e2e-inf-a-e3c4a7-57c4dc9874-swj7w\" (UID: \"2f7b4b6e-9a7c-4ce8-aa0b-600cded35eb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e3c4a7-57c4dc9874-swj7w" Nov 01 20:07:17.295121 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:17.295126 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f7b4b6e-9a7c-4ce8-aa0b-600cded35eb3-metrics-certs\") pod \"maas-api-e2e-inf-a-e3c4a7-57c4dc9874-swj7w\" (UID: \"2f7b4b6e-9a7c-4ce8-aa0b-600cded35eb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e3c4a7-57c4dc9874-swj7w" Nov 01 20:07:17.295350 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:17.295159 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f7b4b6e-9a7c-4ce8-aa0b-600cded35eb3-maas-api-tls\") pod \"maas-api-e2e-inf-a-e3c4a7-57c4dc9874-swj7w\" (UID: \"2f7b4b6e-9a7c-4ce8-aa0b-600cded35eb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e3c4a7-57c4dc9874-swj7w" Nov 01 20:07:17.295350 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:17.295328 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f7b4b6e-9a7c-4ce8-aa0b-600cded35eb3-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e3c4a7-57c4dc9874-swj7w\" (UID: \"2f7b4b6e-9a7c-4ce8-aa0b-600cded35eb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e3c4a7-57c4dc9874-swj7w" Nov 01 20:07:17.396598 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:17.396514 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f7b4b6e-9a7c-4ce8-aa0b-600cded35eb3-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e3c4a7-57c4dc9874-swj7w\" (UID: \"2f7b4b6e-9a7c-4ce8-aa0b-600cded35eb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e3c4a7-57c4dc9874-swj7w" Nov 01 20:07:17.396750 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:17.396627 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtbdw\" (UniqueName: \"kubernetes.io/projected/2f7b4b6e-9a7c-4ce8-aa0b-600cded35eb3-kube-api-access-wtbdw\") pod \"maas-api-e2e-inf-a-e3c4a7-57c4dc9874-swj7w\" (UID: \"2f7b4b6e-9a7c-4ce8-aa0b-600cded35eb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e3c4a7-57c4dc9874-swj7w" Nov 01 20:07:17.396750 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:17.396657 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f7b4b6e-9a7c-4ce8-aa0b-600cded35eb3-metrics-certs\") pod \"maas-api-e2e-inf-a-e3c4a7-57c4dc9874-swj7w\" (UID: \"2f7b4b6e-9a7c-4ce8-aa0b-600cded35eb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e3c4a7-57c4dc9874-swj7w" Nov 01 20:07:17.396836 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:17.396780 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f7b4b6e-9a7c-4ce8-aa0b-600cded35eb3-maas-api-tls\") pod \"maas-api-e2e-inf-a-e3c4a7-57c4dc9874-swj7w\" (UID: \"2f7b4b6e-9a7c-4ce8-aa0b-600cded35eb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e3c4a7-57c4dc9874-swj7w" Nov 01 20:07:17.397237 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:17.397205 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f7b4b6e-9a7c-4ce8-aa0b-600cded35eb3-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e3c4a7-57c4dc9874-swj7w\" (UID: \"2f7b4b6e-9a7c-4ce8-aa0b-600cded35eb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e3c4a7-57c4dc9874-swj7w" Nov 01 20:07:17.399156 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:17.399130 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f7b4b6e-9a7c-4ce8-aa0b-600cded35eb3-maas-api-tls\") pod \"maas-api-e2e-inf-a-e3c4a7-57c4dc9874-swj7w\" (UID: \"2f7b4b6e-9a7c-4ce8-aa0b-600cded35eb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e3c4a7-57c4dc9874-swj7w" Nov 01 20:07:17.399307 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:17.399272 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f7b4b6e-9a7c-4ce8-aa0b-600cded35eb3-metrics-certs\") pod \"maas-api-e2e-inf-a-e3c4a7-57c4dc9874-swj7w\" (UID: \"2f7b4b6e-9a7c-4ce8-aa0b-600cded35eb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e3c4a7-57c4dc9874-swj7w" Nov 01 20:07:17.407494 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:17.407468 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtbdw\" (UniqueName: \"kubernetes.io/projected/2f7b4b6e-9a7c-4ce8-aa0b-600cded35eb3-kube-api-access-wtbdw\") pod \"maas-api-e2e-inf-a-e3c4a7-57c4dc9874-swj7w\" (UID: \"2f7b4b6e-9a7c-4ce8-aa0b-600cded35eb3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e3c4a7-57c4dc9874-swj7w" Nov 01 20:07:17.466857 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:17.466830 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e3c4a7-57c4dc9874-swj7w" Nov 01 20:07:17.743474 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:17.743447 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e3c4a7-57c4dc9874-swj7w"] Nov 01 20:07:17.745339 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:07:17.745311 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f7b4b6e_9a7c_4ce8_aa0b_600cded35eb3.slice/crio-a7ebe7d20825a298517584f1004db817ac9e14c7b487ce384bc64e0cbb545615 WatchSource:0}: Error finding container a7ebe7d20825a298517584f1004db817ac9e14c7b487ce384bc64e0cbb545615: Status 404 returned error can't find the container with id a7ebe7d20825a298517584f1004db817ac9e14c7b487ce384bc64e0cbb545615 Nov 01 20:07:17.959876 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:17.959840 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e3c4a7-57c4dc9874-swj7w" event={"ID":"2f7b4b6e-9a7c-4ce8-aa0b-600cded35eb3","Type":"ContainerStarted","Data":"a7ebe7d20825a298517584f1004db817ac9e14c7b487ce384bc64e0cbb545615"} Nov 01 20:07:18.966770 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:18.966736 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e3c4a7-57c4dc9874-swj7w" event={"ID":"2f7b4b6e-9a7c-4ce8-aa0b-600cded35eb3","Type":"ContainerStarted","Data":"e83d858f53c7d1c0013d1e6c9fb0aefcea45bb727c7700354c116460b3c23ee5"} Nov 01 20:07:18.967188 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:18.966805 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e3c4a7-57c4dc9874-swj7w" Nov 01 20:07:18.985709 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:18.985655 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e3c4a7-57c4dc9874-swj7w" podStartSLOduration=1.58458786 podStartE2EDuration="1.985641552s" totalImagesPullingTime="401.053692ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:07:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:07:18.983965468 +0000 UTC m=+484.757125976" watchObservedRunningTime="2026-11-01 20:07:18.985641552 +0000 UTC m=+484.758802066" Nov 01 20:07:24.977740 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:24.977714 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e3c4a7-57c4dc9874-swj7w" Nov 01 20:07:35.842824 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:07:35.842784 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:07:40.971261 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:40.971230 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56"] Nov 01 20:07:40.975165 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:40.975149 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56" Nov 01 20:07:40.985082 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:40.985057 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56"] Nov 01 20:07:41.017333 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:41.017300 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-3a6480-openshift-default-dockercfg-7wlx7\"" Nov 01 20:07:41.018636 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:41.018612 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe8488d-838d-47fe-b2c5-8b88dfb5724e-workload-socket\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56\" (UID: \"1fe8488d-838d-47fe-b2c5-8b88dfb5724e\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56" Nov 01 20:07:41.018782 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:41.018654 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-885hx\" (UniqueName: \"kubernetes.io/projected/1fe8488d-838d-47fe-b2c5-8b88dfb5724e-kube-api-access-885hx\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56\" (UID: \"1fe8488d-838d-47fe-b2c5-8b88dfb5724e\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56" Nov 01 20:07:41.018782 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:41.018675 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fe8488d-838d-47fe-b2c5-8b88dfb5724e-istiod-ca-cert\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56\" (UID: \"1fe8488d-838d-47fe-b2c5-8b88dfb5724e\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56" Nov 01 20:07:41.018782 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:41.018699 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fe8488d-838d-47fe-b2c5-8b88dfb5724e-istio-envoy\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56\" (UID: \"1fe8488d-838d-47fe-b2c5-8b88dfb5724e\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56" Nov 01 20:07:41.018782 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:41.018724 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fe8488d-838d-47fe-b2c5-8b88dfb5724e-istio-token\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56\" (UID: \"1fe8488d-838d-47fe-b2c5-8b88dfb5724e\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56" Nov 01 20:07:41.018782 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:41.018766 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fe8488d-838d-47fe-b2c5-8b88dfb5724e-istio-data\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56\" (UID: \"1fe8488d-838d-47fe-b2c5-8b88dfb5724e\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56" Nov 01 20:07:41.019098 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:41.019073 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fe8488d-838d-47fe-b2c5-8b88dfb5724e-istio-podinfo\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56\" (UID: \"1fe8488d-838d-47fe-b2c5-8b88dfb5724e\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56" Nov 01 20:07:41.019276 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:41.019247 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe8488d-838d-47fe-b2c5-8b88dfb5724e-credential-socket\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56\" (UID: \"1fe8488d-838d-47fe-b2c5-8b88dfb5724e\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56" Nov 01 20:07:41.019470 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:41.019330 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fe8488d-838d-47fe-b2c5-8b88dfb5724e-workload-certs\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56\" (UID: \"1fe8488d-838d-47fe-b2c5-8b88dfb5724e\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56" Nov 01 20:07:41.120739 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:41.120705 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fe8488d-838d-47fe-b2c5-8b88dfb5724e-istio-token\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56\" (UID: \"1fe8488d-838d-47fe-b2c5-8b88dfb5724e\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56" Nov 01 20:07:41.120915 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:41.120764 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fe8488d-838d-47fe-b2c5-8b88dfb5724e-istio-data\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56\" (UID: \"1fe8488d-838d-47fe-b2c5-8b88dfb5724e\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56" Nov 01 20:07:41.120915 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:41.120810 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fe8488d-838d-47fe-b2c5-8b88dfb5724e-istio-podinfo\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56\" (UID: \"1fe8488d-838d-47fe-b2c5-8b88dfb5724e\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56" Nov 01 20:07:41.120915 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:41.120863 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe8488d-838d-47fe-b2c5-8b88dfb5724e-credential-socket\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56\" (UID: \"1fe8488d-838d-47fe-b2c5-8b88dfb5724e\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56" Nov 01 20:07:41.121086 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:41.120920 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fe8488d-838d-47fe-b2c5-8b88dfb5724e-workload-certs\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56\" (UID: \"1fe8488d-838d-47fe-b2c5-8b88dfb5724e\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56" Nov 01 20:07:41.121086 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:41.120967 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe8488d-838d-47fe-b2c5-8b88dfb5724e-workload-socket\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56\" (UID: \"1fe8488d-838d-47fe-b2c5-8b88dfb5724e\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56" Nov 01 20:07:41.121185 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:41.121093 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-885hx\" (UniqueName: \"kubernetes.io/projected/1fe8488d-838d-47fe-b2c5-8b88dfb5724e-kube-api-access-885hx\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56\" (UID: \"1fe8488d-838d-47fe-b2c5-8b88dfb5724e\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56" Nov 01 20:07:41.121185 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:41.121137 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fe8488d-838d-47fe-b2c5-8b88dfb5724e-istiod-ca-cert\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56\" (UID: \"1fe8488d-838d-47fe-b2c5-8b88dfb5724e\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56" Nov 01 20:07:41.121185 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:41.121141 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fe8488d-838d-47fe-b2c5-8b88dfb5724e-istio-data\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56\" (UID: \"1fe8488d-838d-47fe-b2c5-8b88dfb5724e\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56" Nov 01 20:07:41.121185 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:41.121175 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fe8488d-838d-47fe-b2c5-8b88dfb5724e-istio-envoy\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56\" (UID: \"1fe8488d-838d-47fe-b2c5-8b88dfb5724e\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56" Nov 01 20:07:41.121416 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:41.121241 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe8488d-838d-47fe-b2c5-8b88dfb5724e-credential-socket\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56\" (UID: \"1fe8488d-838d-47fe-b2c5-8b88dfb5724e\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56" Nov 01 20:07:41.121571 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:41.121547 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe8488d-838d-47fe-b2c5-8b88dfb5724e-workload-socket\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56\" (UID: \"1fe8488d-838d-47fe-b2c5-8b88dfb5724e\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56" Nov 01 20:07:41.121644 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:41.121576 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fe8488d-838d-47fe-b2c5-8b88dfb5724e-workload-certs\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56\" (UID: \"1fe8488d-838d-47fe-b2c5-8b88dfb5724e\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56" Nov 01 20:07:41.121774 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:41.121752 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fe8488d-838d-47fe-b2c5-8b88dfb5724e-istiod-ca-cert\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56\" (UID: \"1fe8488d-838d-47fe-b2c5-8b88dfb5724e\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56" Nov 01 20:07:41.123306 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:41.123276 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fe8488d-838d-47fe-b2c5-8b88dfb5724e-istio-envoy\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56\" (UID: \"1fe8488d-838d-47fe-b2c5-8b88dfb5724e\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56" Nov 01 20:07:41.123530 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:41.123515 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fe8488d-838d-47fe-b2c5-8b88dfb5724e-istio-podinfo\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56\" (UID: \"1fe8488d-838d-47fe-b2c5-8b88dfb5724e\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56" Nov 01 20:07:41.128692 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:41.128665 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-885hx\" (UniqueName: \"kubernetes.io/projected/1fe8488d-838d-47fe-b2c5-8b88dfb5724e-kube-api-access-885hx\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56\" (UID: \"1fe8488d-838d-47fe-b2c5-8b88dfb5724e\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56" Nov 01 20:07:41.128982 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:41.128961 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fe8488d-838d-47fe-b2c5-8b88dfb5724e-istio-token\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56\" (UID: \"1fe8488d-838d-47fe-b2c5-8b88dfb5724e\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56" Nov 01 20:07:41.290229 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:41.290137 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56" Nov 01 20:07:41.579396 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:41.579374 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56"] Nov 01 20:07:41.581103 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:07:41.581071 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fe8488d_838d_47fe_b2c5_8b88dfb5724e.slice/crio-c17e489a958b11ee57d45474d695617f9745bcba898aef4e69f4801cead71d4f WatchSource:0}: Error finding container c17e489a958b11ee57d45474d695617f9745bcba898aef4e69f4801cead71d4f: Status 404 returned error can't find the container with id c17e489a958b11ee57d45474d695617f9745bcba898aef4e69f4801cead71d4f Nov 01 20:07:41.583510 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:41.583477 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:07:41.583582 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:41.583547 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:07:41.583623 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:41.583579 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:07:42.092795 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:42.092761 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56" event={"ID":"1fe8488d-838d-47fe-b2c5-8b88dfb5724e","Type":"ContainerStarted","Data":"122796766cd7ef50fe4d4c7a2ace79ee85d0bb38c329595b5333b9553653dd49"} Nov 01 20:07:42.093271 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:42.092811 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56" event={"ID":"1fe8488d-838d-47fe-b2c5-8b88dfb5724e","Type":"ContainerStarted","Data":"c17e489a958b11ee57d45474d695617f9745bcba898aef4e69f4801cead71d4f"} Nov 01 20:07:42.148640 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:42.148589 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56" podStartSLOduration=2.148573275 podStartE2EDuration="2.148573275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:07:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:07:42.147338648 +0000 UTC m=+507.920499161" watchObservedRunningTime="2026-11-01 20:07:42.148573275 +0000 UTC m=+507.921733786" Nov 01 20:07:42.290521 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:42.290489 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56" Nov 01 20:07:42.292142 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:42.292114 5935 patch_prober.go:29] interesting pod/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Nov 01 20:07:42.292258 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:42.292164 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56" podUID="1fe8488d-838d-47fe-b2c5-8b88dfb5724e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Nov 01 20:07:43.290954 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:43.290922 5935 patch_prober.go:29] interesting pod/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Nov 01 20:07:43.291397 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:43.290972 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56" podUID="1fe8488d-838d-47fe-b2c5-8b88dfb5724e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Nov 01 20:07:44.290631 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:44.290595 5935 patch_prober.go:29] interesting pod/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Nov 01 20:07:44.290811 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:44.290648 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56" podUID="1fe8488d-838d-47fe-b2c5-8b88dfb5724e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Nov 01 20:07:45.290996 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:45.290956 5935 patch_prober.go:29] interesting pod/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Nov 01 20:07:45.291382 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:45.291018 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56" podUID="1fe8488d-838d-47fe-b2c5-8b88dfb5724e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Nov 01 20:07:46.180063 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:46.180029 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-0aa61b/distinct-w4-0aa61b-kserve-6674df959d-2lm95"] Nov 01 20:07:46.183880 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:46.183858 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-w4-0aa61b-kserve-6674df959d-2lm95" Nov 01 20:07:46.190402 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:46.190377 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-0aa61b\"/\"distinct-w4-0aa61b-kserve-self-signed-certs\"" Nov 01 20:07:46.191692 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:46.191667 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-0aa61b/distinct-w4-0aa61b-kserve-6674df959d-2lm95"] Nov 01 20:07:46.290766 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:46.290735 5935 patch_prober.go:29] interesting pod/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Nov 01 20:07:46.290989 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:46.290969 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56" podUID="1fe8488d-838d-47fe-b2c5-8b88dfb5724e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Nov 01 20:07:46.294878 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:46.294856 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7f35703-1219-4dd4-8bbd-631fb7303724-tmp-dir\") pod \"distinct-w4-0aa61b-kserve-6674df959d-2lm95\" (UID: \"b7f35703-1219-4dd4-8bbd-631fb7303724\") " pod="e2e-models-e2e-worker-w4-0aa61b/distinct-w4-0aa61b-kserve-6674df959d-2lm95" Nov 01 20:07:46.295312 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:46.295272 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7f35703-1219-4dd4-8bbd-631fb7303724-tls-certs\") pod \"distinct-w4-0aa61b-kserve-6674df959d-2lm95\" (UID: \"b7f35703-1219-4dd4-8bbd-631fb7303724\") " pod="e2e-models-e2e-worker-w4-0aa61b/distinct-w4-0aa61b-kserve-6674df959d-2lm95" Nov 01 20:07:46.295444 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:46.295428 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5z2t\" (UniqueName: \"kubernetes.io/projected/b7f35703-1219-4dd4-8bbd-631fb7303724-kube-api-access-f5z2t\") pod \"distinct-w4-0aa61b-kserve-6674df959d-2lm95\" (UID: \"b7f35703-1219-4dd4-8bbd-631fb7303724\") " pod="e2e-models-e2e-worker-w4-0aa61b/distinct-w4-0aa61b-kserve-6674df959d-2lm95" Nov 01 20:07:46.295591 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:46.295564 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7f35703-1219-4dd4-8bbd-631fb7303724-home\") pod \"distinct-w4-0aa61b-kserve-6674df959d-2lm95\" (UID: \"b7f35703-1219-4dd4-8bbd-631fb7303724\") " pod="e2e-models-e2e-worker-w4-0aa61b/distinct-w4-0aa61b-kserve-6674df959d-2lm95" Nov 01 20:07:46.295672 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:46.295615 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7f35703-1219-4dd4-8bbd-631fb7303724-model-cache\") pod \"distinct-w4-0aa61b-kserve-6674df959d-2lm95\" (UID: \"b7f35703-1219-4dd4-8bbd-631fb7303724\") " pod="e2e-models-e2e-worker-w4-0aa61b/distinct-w4-0aa61b-kserve-6674df959d-2lm95" Nov 01 20:07:46.295672 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:46.295663 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7f35703-1219-4dd4-8bbd-631fb7303724-dshm\") pod \"distinct-w4-0aa61b-kserve-6674df959d-2lm95\" (UID: \"b7f35703-1219-4dd4-8bbd-631fb7303724\") " pod="e2e-models-e2e-worker-w4-0aa61b/distinct-w4-0aa61b-kserve-6674df959d-2lm95" Nov 01 20:07:46.396719 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:46.396682 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7f35703-1219-4dd4-8bbd-631fb7303724-dshm\") pod \"distinct-w4-0aa61b-kserve-6674df959d-2lm95\" (UID: \"b7f35703-1219-4dd4-8bbd-631fb7303724\") " pod="e2e-models-e2e-worker-w4-0aa61b/distinct-w4-0aa61b-kserve-6674df959d-2lm95" Nov 01 20:07:46.396885 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:46.396763 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7f35703-1219-4dd4-8bbd-631fb7303724-tmp-dir\") pod \"distinct-w4-0aa61b-kserve-6674df959d-2lm95\" (UID: \"b7f35703-1219-4dd4-8bbd-631fb7303724\") " pod="e2e-models-e2e-worker-w4-0aa61b/distinct-w4-0aa61b-kserve-6674df959d-2lm95" Nov 01 20:07:46.396885 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:46.396818 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7f35703-1219-4dd4-8bbd-631fb7303724-tls-certs\") pod \"distinct-w4-0aa61b-kserve-6674df959d-2lm95\" (UID: \"b7f35703-1219-4dd4-8bbd-631fb7303724\") " pod="e2e-models-e2e-worker-w4-0aa61b/distinct-w4-0aa61b-kserve-6674df959d-2lm95" Nov 01 20:07:46.396885 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:46.396842 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5z2t\" (UniqueName: \"kubernetes.io/projected/b7f35703-1219-4dd4-8bbd-631fb7303724-kube-api-access-f5z2t\") pod \"distinct-w4-0aa61b-kserve-6674df959d-2lm95\" (UID: \"b7f35703-1219-4dd4-8bbd-631fb7303724\") " pod="e2e-models-e2e-worker-w4-0aa61b/distinct-w4-0aa61b-kserve-6674df959d-2lm95" Nov 01 20:07:46.396885 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:46.396878 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7f35703-1219-4dd4-8bbd-631fb7303724-home\") pod \"distinct-w4-0aa61b-kserve-6674df959d-2lm95\" (UID: \"b7f35703-1219-4dd4-8bbd-631fb7303724\") " pod="e2e-models-e2e-worker-w4-0aa61b/distinct-w4-0aa61b-kserve-6674df959d-2lm95" Nov 01 20:07:46.397017 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:46.396903 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7f35703-1219-4dd4-8bbd-631fb7303724-model-cache\") pod \"distinct-w4-0aa61b-kserve-6674df959d-2lm95\" (UID: \"b7f35703-1219-4dd4-8bbd-631fb7303724\") " pod="e2e-models-e2e-worker-w4-0aa61b/distinct-w4-0aa61b-kserve-6674df959d-2lm95" Nov 01 20:07:46.397281 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:46.397264 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7f35703-1219-4dd4-8bbd-631fb7303724-model-cache\") pod \"distinct-w4-0aa61b-kserve-6674df959d-2lm95\" (UID: \"b7f35703-1219-4dd4-8bbd-631fb7303724\") " pod="e2e-models-e2e-worker-w4-0aa61b/distinct-w4-0aa61b-kserve-6674df959d-2lm95" Nov 01 20:07:46.398035 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:46.398010 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7f35703-1219-4dd4-8bbd-631fb7303724-tmp-dir\") pod \"distinct-w4-0aa61b-kserve-6674df959d-2lm95\" (UID: \"b7f35703-1219-4dd4-8bbd-631fb7303724\") " pod="e2e-models-e2e-worker-w4-0aa61b/distinct-w4-0aa61b-kserve-6674df959d-2lm95" Nov 01 20:07:46.398449 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:46.398428 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7f35703-1219-4dd4-8bbd-631fb7303724-home\") pod \"distinct-w4-0aa61b-kserve-6674df959d-2lm95\" (UID: \"b7f35703-1219-4dd4-8bbd-631fb7303724\") " pod="e2e-models-e2e-worker-w4-0aa61b/distinct-w4-0aa61b-kserve-6674df959d-2lm95" Nov 01 20:07:46.399473 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:46.399451 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7f35703-1219-4dd4-8bbd-631fb7303724-dshm\") pod \"distinct-w4-0aa61b-kserve-6674df959d-2lm95\" (UID: \"b7f35703-1219-4dd4-8bbd-631fb7303724\") " pod="e2e-models-e2e-worker-w4-0aa61b/distinct-w4-0aa61b-kserve-6674df959d-2lm95" Nov 01 20:07:46.400449 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:46.400428 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7f35703-1219-4dd4-8bbd-631fb7303724-tls-certs\") pod \"distinct-w4-0aa61b-kserve-6674df959d-2lm95\" (UID: \"b7f35703-1219-4dd4-8bbd-631fb7303724\") " pod="e2e-models-e2e-worker-w4-0aa61b/distinct-w4-0aa61b-kserve-6674df959d-2lm95" Nov 01 20:07:46.407363 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:46.407343 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5z2t\" (UniqueName: \"kubernetes.io/projected/b7f35703-1219-4dd4-8bbd-631fb7303724-kube-api-access-f5z2t\") pod \"distinct-w4-0aa61b-kserve-6674df959d-2lm95\" (UID: \"b7f35703-1219-4dd4-8bbd-631fb7303724\") " pod="e2e-models-e2e-worker-w4-0aa61b/distinct-w4-0aa61b-kserve-6674df959d-2lm95" Nov 01 20:07:46.500278 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:46.500247 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-w4-0aa61b-kserve-6674df959d-2lm95" Nov 01 20:07:46.809251 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:46.809205 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-0aa61b/distinct-w4-0aa61b-kserve-6674df959d-2lm95"] Nov 01 20:07:46.812689 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:07:46.812658 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb7f35703_1219_4dd4_8bbd_631fb7303724.slice/crio-985e8f26fd2c49a598f616f622ecd3f8bffabdff88d04705c1aaf1c3d30fd15f WatchSource:0}: Error finding container 985e8f26fd2c49a598f616f622ecd3f8bffabdff88d04705c1aaf1c3d30fd15f: Status 404 returned error can't find the container with id 985e8f26fd2c49a598f616f622ecd3f8bffabdff88d04705c1aaf1c3d30fd15f Nov 01 20:07:47.120134 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:47.120036 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-w4-0aa61b-kserve-6674df959d-2lm95" event={"ID":"b7f35703-1219-4dd4-8bbd-631fb7303724","Type":"ContainerStarted","Data":"06c8037c5eb2db162c520d833052fd0413f5144056484e4a21e10af771e3c1b2"} Nov 01 20:07:47.120134 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:47.120086 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-w4-0aa61b-kserve-6674df959d-2lm95" event={"ID":"b7f35703-1219-4dd4-8bbd-631fb7303724","Type":"ContainerStarted","Data":"985e8f26fd2c49a598f616f622ecd3f8bffabdff88d04705c1aaf1c3d30fd15f"} Nov 01 20:07:47.137115 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:47.137065 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-w4-0aa61b-kserve-6674df959d-2lm95" podStartSLOduration=1.13705093 podStartE2EDuration="1.13705093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:07:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:07:47.136336053 +0000 UTC m=+512.909496566" watchObservedRunningTime="2026-11-01 20:07:47.13705093 +0000 UTC m=+512.910211442" Nov 01 20:07:47.291328 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:47.290683 5935 patch_prober.go:29] interesting pod/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Nov 01 20:07:47.291328 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:47.290753 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56" podUID="1fe8488d-838d-47fe-b2c5-8b88dfb5724e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Nov 01 20:07:47.500537 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:47.500499 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-w4-0aa61b-kserve-6674df959d-2lm95" Nov 01 20:07:48.291068 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:48.291034 5935 patch_prober.go:29] interesting pod/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Nov 01 20:07:48.291239 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:48.291088 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56" podUID="1fe8488d-838d-47fe-b2c5-8b88dfb5724e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Nov 01 20:07:49.301751 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:49.301711 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56" Nov 01 20:07:49.302354 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:49.302181 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56" Nov 01 20:07:49.303377 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:49.303348 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56" Nov 01 20:07:55.179630 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:55.179594 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-e3c4a7/test-model-e3c4a7-kserve-59c98dffb5-cqbbk"] Nov 01 20:07:55.185048 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:55.185024 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e3c4a7/test-model-e3c4a7-kserve-59c98dffb5-cqbbk" Nov 01 20:07:55.190745 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:55.190716 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-e3c4a7\"/\"default-dockercfg-xptcn\"" Nov 01 20:07:55.190855 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:55.190764 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-e3c4a7\"/\"test-model-e3c4a7-kserve-self-signed-certs\"" Nov 01 20:07:55.190855 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:55.190785 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-e3c4a7\"/\"kube-root-ca.crt\"" Nov 01 20:07:55.191955 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:55.191932 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-e3c4a7/test-model-e3c4a7-kserve-59c98dffb5-cqbbk"] Nov 01 20:07:55.192064 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:55.192008 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-e3c4a7\"/\"openshift-service-ca.crt\"" Nov 01 20:07:55.286364 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:55.286321 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5df4b03-d387-44c7-a263-e24f9730ed0e-model-cache\") pod \"test-model-e3c4a7-kserve-59c98dffb5-cqbbk\" (UID: \"b5df4b03-d387-44c7-a263-e24f9730ed0e\") " pod="ai-tenant-e2e-inf-a-e3c4a7/test-model-e3c4a7-kserve-59c98dffb5-cqbbk" Nov 01 20:07:55.286523 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:55.286454 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5df4b03-d387-44c7-a263-e24f9730ed0e-tls-certs\") pod \"test-model-e3c4a7-kserve-59c98dffb5-cqbbk\" (UID: \"b5df4b03-d387-44c7-a263-e24f9730ed0e\") " pod="ai-tenant-e2e-inf-a-e3c4a7/test-model-e3c4a7-kserve-59c98dffb5-cqbbk" Nov 01 20:07:55.286523 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:55.286504 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5df4b03-d387-44c7-a263-e24f9730ed0e-home\") pod \"test-model-e3c4a7-kserve-59c98dffb5-cqbbk\" (UID: \"b5df4b03-d387-44c7-a263-e24f9730ed0e\") " pod="ai-tenant-e2e-inf-a-e3c4a7/test-model-e3c4a7-kserve-59c98dffb5-cqbbk" Nov 01 20:07:55.286691 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:55.286666 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksdfx\" (UniqueName: \"kubernetes.io/projected/b5df4b03-d387-44c7-a263-e24f9730ed0e-kube-api-access-ksdfx\") pod \"test-model-e3c4a7-kserve-59c98dffb5-cqbbk\" (UID: \"b5df4b03-d387-44c7-a263-e24f9730ed0e\") " pod="ai-tenant-e2e-inf-a-e3c4a7/test-model-e3c4a7-kserve-59c98dffb5-cqbbk" Nov 01 20:07:55.286744 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:55.286725 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5df4b03-d387-44c7-a263-e24f9730ed0e-tmp-dir\") pod \"test-model-e3c4a7-kserve-59c98dffb5-cqbbk\" (UID: \"b5df4b03-d387-44c7-a263-e24f9730ed0e\") " pod="ai-tenant-e2e-inf-a-e3c4a7/test-model-e3c4a7-kserve-59c98dffb5-cqbbk" Nov 01 20:07:55.286778 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:55.286768 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5df4b03-d387-44c7-a263-e24f9730ed0e-dshm\") pod \"test-model-e3c4a7-kserve-59c98dffb5-cqbbk\" (UID: \"b5df4b03-d387-44c7-a263-e24f9730ed0e\") " pod="ai-tenant-e2e-inf-a-e3c4a7/test-model-e3c4a7-kserve-59c98dffb5-cqbbk" Nov 01 20:07:55.387567 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:55.387527 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5df4b03-d387-44c7-a263-e24f9730ed0e-dshm\") pod \"test-model-e3c4a7-kserve-59c98dffb5-cqbbk\" (UID: \"b5df4b03-d387-44c7-a263-e24f9730ed0e\") " pod="ai-tenant-e2e-inf-a-e3c4a7/test-model-e3c4a7-kserve-59c98dffb5-cqbbk" Nov 01 20:07:55.387901 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:55.387873 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5df4b03-d387-44c7-a263-e24f9730ed0e-model-cache\") pod \"test-model-e3c4a7-kserve-59c98dffb5-cqbbk\" (UID: \"b5df4b03-d387-44c7-a263-e24f9730ed0e\") " pod="ai-tenant-e2e-inf-a-e3c4a7/test-model-e3c4a7-kserve-59c98dffb5-cqbbk" Nov 01 20:07:55.388157 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:55.388137 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5df4b03-d387-44c7-a263-e24f9730ed0e-tls-certs\") pod \"test-model-e3c4a7-kserve-59c98dffb5-cqbbk\" (UID: \"b5df4b03-d387-44c7-a263-e24f9730ed0e\") " pod="ai-tenant-e2e-inf-a-e3c4a7/test-model-e3c4a7-kserve-59c98dffb5-cqbbk" Nov 01 20:07:55.388305 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:55.388274 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5df4b03-d387-44c7-a263-e24f9730ed0e-home\") pod \"test-model-e3c4a7-kserve-59c98dffb5-cqbbk\" (UID: \"b5df4b03-d387-44c7-a263-e24f9730ed0e\") " pod="ai-tenant-e2e-inf-a-e3c4a7/test-model-e3c4a7-kserve-59c98dffb5-cqbbk" Nov 01 20:07:55.388463 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:55.388448 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksdfx\" (UniqueName: \"kubernetes.io/projected/b5df4b03-d387-44c7-a263-e24f9730ed0e-kube-api-access-ksdfx\") pod \"test-model-e3c4a7-kserve-59c98dffb5-cqbbk\" (UID: \"b5df4b03-d387-44c7-a263-e24f9730ed0e\") " pod="ai-tenant-e2e-inf-a-e3c4a7/test-model-e3c4a7-kserve-59c98dffb5-cqbbk" Nov 01 20:07:55.388593 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:55.388578 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5df4b03-d387-44c7-a263-e24f9730ed0e-tmp-dir\") pod \"test-model-e3c4a7-kserve-59c98dffb5-cqbbk\" (UID: \"b5df4b03-d387-44c7-a263-e24f9730ed0e\") " pod="ai-tenant-e2e-inf-a-e3c4a7/test-model-e3c4a7-kserve-59c98dffb5-cqbbk" Nov 01 20:07:55.389001 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:55.388975 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5df4b03-d387-44c7-a263-e24f9730ed0e-tmp-dir\") pod \"test-model-e3c4a7-kserve-59c98dffb5-cqbbk\" (UID: \"b5df4b03-d387-44c7-a263-e24f9730ed0e\") " pod="ai-tenant-e2e-inf-a-e3c4a7/test-model-e3c4a7-kserve-59c98dffb5-cqbbk" Nov 01 20:07:55.389250 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:55.389230 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5df4b03-d387-44c7-a263-e24f9730ed0e-home\") pod \"test-model-e3c4a7-kserve-59c98dffb5-cqbbk\" (UID: \"b5df4b03-d387-44c7-a263-e24f9730ed0e\") " pod="ai-tenant-e2e-inf-a-e3c4a7/test-model-e3c4a7-kserve-59c98dffb5-cqbbk" Nov 01 20:07:55.389359 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:55.388303 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5df4b03-d387-44c7-a263-e24f9730ed0e-model-cache\") pod \"test-model-e3c4a7-kserve-59c98dffb5-cqbbk\" (UID: \"b5df4b03-d387-44c7-a263-e24f9730ed0e\") " pod="ai-tenant-e2e-inf-a-e3c4a7/test-model-e3c4a7-kserve-59c98dffb5-cqbbk" Nov 01 20:07:55.391874 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:55.391850 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5df4b03-d387-44c7-a263-e24f9730ed0e-tls-certs\") pod \"test-model-e3c4a7-kserve-59c98dffb5-cqbbk\" (UID: \"b5df4b03-d387-44c7-a263-e24f9730ed0e\") " pod="ai-tenant-e2e-inf-a-e3c4a7/test-model-e3c4a7-kserve-59c98dffb5-cqbbk" Nov 01 20:07:55.392162 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:55.392136 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5df4b03-d387-44c7-a263-e24f9730ed0e-dshm\") pod \"test-model-e3c4a7-kserve-59c98dffb5-cqbbk\" (UID: \"b5df4b03-d387-44c7-a263-e24f9730ed0e\") " pod="ai-tenant-e2e-inf-a-e3c4a7/test-model-e3c4a7-kserve-59c98dffb5-cqbbk" Nov 01 20:07:55.399779 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:55.399758 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksdfx\" (UniqueName: \"kubernetes.io/projected/b5df4b03-d387-44c7-a263-e24f9730ed0e-kube-api-access-ksdfx\") pod \"test-model-e3c4a7-kserve-59c98dffb5-cqbbk\" (UID: \"b5df4b03-d387-44c7-a263-e24f9730ed0e\") " pod="ai-tenant-e2e-inf-a-e3c4a7/test-model-e3c4a7-kserve-59c98dffb5-cqbbk" Nov 01 20:07:55.499531 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:55.499500 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e3c4a7/test-model-e3c4a7-kserve-59c98dffb5-cqbbk" Nov 01 20:07:55.782527 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:55.782492 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-e3c4a7/test-model-e3c4a7-kserve-59c98dffb5-cqbbk"] Nov 01 20:07:55.784836 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:07:55.784794 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb5df4b03_d387_44c7_a263_e24f9730ed0e.slice/crio-ff44e0192dcdcb0b535dfe9401b96fbcc4e3b1b8fdd346baeb9265c2fb196946 WatchSource:0}: Error finding container ff44e0192dcdcb0b535dfe9401b96fbcc4e3b1b8fdd346baeb9265c2fb196946: Status 404 returned error can't find the container with id ff44e0192dcdcb0b535dfe9401b96fbcc4e3b1b8fdd346baeb9265c2fb196946 Nov 01 20:07:56.178268 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:56.178182 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e3c4a7/test-model-e3c4a7-kserve-59c98dffb5-cqbbk" event={"ID":"b5df4b03-d387-44c7-a263-e24f9730ed0e","Type":"ContainerStarted","Data":"ab4d91a5764bbfd36be6abe1c4f2f822b91d77ac2390181325a5618ec7a581ea"} Nov 01 20:07:56.178268 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:56.178225 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e3c4a7/test-model-e3c4a7-kserve-59c98dffb5-cqbbk" event={"ID":"b5df4b03-d387-44c7-a263-e24f9730ed0e","Type":"ContainerStarted","Data":"ff44e0192dcdcb0b535dfe9401b96fbcc4e3b1b8fdd346baeb9265c2fb196946"} Nov 01 20:07:56.194859 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:56.194814 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-e3c4a7/test-model-e3c4a7-kserve-59c98dffb5-cqbbk" podStartSLOduration=1.194800276 podStartE2EDuration="1.194800276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:07:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:07:56.193436672 +0000 UTC m=+521.966597183" watchObservedRunningTime="2026-11-01 20:07:56.194800276 +0000 UTC m=+521.967960788" Nov 01 20:07:56.499577 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:56.499551 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-e3c4a7/test-model-e3c4a7-kserve-59c98dffb5-cqbbk" Nov 01 20:07:56.501100 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:56.501078 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-w4-0aa61b-kserve-6674df959d-2lm95" Nov 01 20:07:56.514122 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:56.514100 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-w4-0aa61b-kserve-6674df959d-2lm95" Nov 01 20:07:57.194647 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:07:57.194610 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-w4-0aa61b-kserve-6674df959d-2lm95" Nov 01 20:08:05.499817 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:05.499786 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-e3c4a7/test-model-e3c4a7-kserve-59c98dffb5-cqbbk" Nov 01 20:08:05.513280 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:05.513257 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-e3c4a7/test-model-e3c4a7-kserve-59c98dffb5-cqbbk" Nov 01 20:08:05.843922 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:08:05.843840 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:08:06.252728 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:06.252703 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-e3c4a7/test-model-e3c4a7-kserve-59c98dffb5-cqbbk" Nov 01 20:08:26.128509 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:26.128423 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6"] Nov 01 20:08:26.132256 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:26.132236 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6" Nov 01 20:08:26.144595 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:26.144570 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6"] Nov 01 20:08:26.144809 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:26.144788 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-e27ae7-openshift-default-dockercfg-855jv\"" Nov 01 20:08:26.198077 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:26.198041 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86e63000-fc7e-4429-af13-4046198bca02-istio-token\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6\" (UID: \"86e63000-fc7e-4429-af13-4046198bca02\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6" Nov 01 20:08:26.198240 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:26.198093 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67czq\" (UniqueName: \"kubernetes.io/projected/86e63000-fc7e-4429-af13-4046198bca02-kube-api-access-67czq\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6\" (UID: \"86e63000-fc7e-4429-af13-4046198bca02\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6" Nov 01 20:08:26.198240 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:26.198228 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86e63000-fc7e-4429-af13-4046198bca02-istiod-ca-cert\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6\" (UID: \"86e63000-fc7e-4429-af13-4046198bca02\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6" Nov 01 20:08:26.198361 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:26.198307 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86e63000-fc7e-4429-af13-4046198bca02-istio-data\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6\" (UID: \"86e63000-fc7e-4429-af13-4046198bca02\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6" Nov 01 20:08:26.198361 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:26.198354 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86e63000-fc7e-4429-af13-4046198bca02-credential-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6\" (UID: \"86e63000-fc7e-4429-af13-4046198bca02\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6" Nov 01 20:08:26.198426 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:26.198392 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86e63000-fc7e-4429-af13-4046198bca02-istio-envoy\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6\" (UID: \"86e63000-fc7e-4429-af13-4046198bca02\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6" Nov 01 20:08:26.198461 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:26.198441 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86e63000-fc7e-4429-af13-4046198bca02-istio-podinfo\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6\" (UID: \"86e63000-fc7e-4429-af13-4046198bca02\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6" Nov 01 20:08:26.198495 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:26.198465 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86e63000-fc7e-4429-af13-4046198bca02-workload-certs\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6\" (UID: \"86e63000-fc7e-4429-af13-4046198bca02\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6" Nov 01 20:08:26.198495 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:26.198487 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86e63000-fc7e-4429-af13-4046198bca02-workload-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6\" (UID: \"86e63000-fc7e-4429-af13-4046198bca02\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6" Nov 01 20:08:26.299029 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:26.298996 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86e63000-fc7e-4429-af13-4046198bca02-istio-token\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6\" (UID: \"86e63000-fc7e-4429-af13-4046198bca02\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6" Nov 01 20:08:26.299202 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:26.299039 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67czq\" (UniqueName: \"kubernetes.io/projected/86e63000-fc7e-4429-af13-4046198bca02-kube-api-access-67czq\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6\" (UID: \"86e63000-fc7e-4429-af13-4046198bca02\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6" Nov 01 20:08:26.299202 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:26.299073 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86e63000-fc7e-4429-af13-4046198bca02-istiod-ca-cert\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6\" (UID: \"86e63000-fc7e-4429-af13-4046198bca02\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6" Nov 01 20:08:26.299316 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:26.299245 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86e63000-fc7e-4429-af13-4046198bca02-istio-data\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6\" (UID: \"86e63000-fc7e-4429-af13-4046198bca02\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6" Nov 01 20:08:26.299361 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:26.299313 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86e63000-fc7e-4429-af13-4046198bca02-credential-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6\" (UID: \"86e63000-fc7e-4429-af13-4046198bca02\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6" Nov 01 20:08:26.299413 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:26.299359 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86e63000-fc7e-4429-af13-4046198bca02-istio-envoy\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6\" (UID: \"86e63000-fc7e-4429-af13-4046198bca02\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6" Nov 01 20:08:26.299413 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:26.299405 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86e63000-fc7e-4429-af13-4046198bca02-istio-podinfo\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6\" (UID: \"86e63000-fc7e-4429-af13-4046198bca02\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6" Nov 01 20:08:26.299530 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:26.299436 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86e63000-fc7e-4429-af13-4046198bca02-workload-certs\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6\" (UID: \"86e63000-fc7e-4429-af13-4046198bca02\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6" Nov 01 20:08:26.299588 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:26.299572 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86e63000-fc7e-4429-af13-4046198bca02-workload-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6\" (UID: \"86e63000-fc7e-4429-af13-4046198bca02\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6" Nov 01 20:08:26.299647 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:26.299616 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86e63000-fc7e-4429-af13-4046198bca02-istio-data\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6\" (UID: \"86e63000-fc7e-4429-af13-4046198bca02\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6" Nov 01 20:08:26.299724 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:26.299700 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86e63000-fc7e-4429-af13-4046198bca02-credential-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6\" (UID: \"86e63000-fc7e-4429-af13-4046198bca02\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6" Nov 01 20:08:26.299836 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:26.299818 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86e63000-fc7e-4429-af13-4046198bca02-istiod-ca-cert\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6\" (UID: \"86e63000-fc7e-4429-af13-4046198bca02\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6" Nov 01 20:08:26.300025 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:26.299999 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86e63000-fc7e-4429-af13-4046198bca02-workload-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6\" (UID: \"86e63000-fc7e-4429-af13-4046198bca02\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6" Nov 01 20:08:26.300173 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:26.300078 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86e63000-fc7e-4429-af13-4046198bca02-workload-certs\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6\" (UID: \"86e63000-fc7e-4429-af13-4046198bca02\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6" Nov 01 20:08:26.301793 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:26.301767 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86e63000-fc7e-4429-af13-4046198bca02-istio-envoy\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6\" (UID: \"86e63000-fc7e-4429-af13-4046198bca02\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6" Nov 01 20:08:26.301793 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:26.301784 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86e63000-fc7e-4429-af13-4046198bca02-istio-podinfo\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6\" (UID: \"86e63000-fc7e-4429-af13-4046198bca02\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6" Nov 01 20:08:26.306376 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:26.306352 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67czq\" (UniqueName: \"kubernetes.io/projected/86e63000-fc7e-4429-af13-4046198bca02-kube-api-access-67czq\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6\" (UID: \"86e63000-fc7e-4429-af13-4046198bca02\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6" Nov 01 20:08:26.306480 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:26.306466 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86e63000-fc7e-4429-af13-4046198bca02-istio-token\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6\" (UID: \"86e63000-fc7e-4429-af13-4046198bca02\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6" Nov 01 20:08:26.446403 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:26.446380 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6" Nov 01 20:08:26.722779 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:26.722753 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6"] Nov 01 20:08:26.725219 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:08:26.725192 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86e63000_fc7e_4429_af13_4046198bca02.slice/crio-7c96c1917d1455f7d7b556adae2e0a2a71b804af0a8cdb3350d54dce29ea2a1f WatchSource:0}: Error finding container 7c96c1917d1455f7d7b556adae2e0a2a71b804af0a8cdb3350d54dce29ea2a1f: Status 404 returned error can't find the container with id 7c96c1917d1455f7d7b556adae2e0a2a71b804af0a8cdb3350d54dce29ea2a1f Nov 01 20:08:26.727413 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:26.727378 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:08:26.727533 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:26.727437 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:08:26.727533 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:26.727470 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:08:27.371188 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:27.371150 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6" event={"ID":"86e63000-fc7e-4429-af13-4046198bca02","Type":"ContainerStarted","Data":"12e7c59c07a00198c5a91bebc1dadc8aeef05856ca87046334c3ed2b97dbab6d"} Nov 01 20:08:27.371822 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:27.371215 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6" event={"ID":"86e63000-fc7e-4429-af13-4046198bca02","Type":"ContainerStarted","Data":"7c96c1917d1455f7d7b556adae2e0a2a71b804af0a8cdb3350d54dce29ea2a1f"} Nov 01 20:08:27.390196 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:27.390151 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6" podStartSLOduration=1.390139556 podStartE2EDuration="1.390139556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:08:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:08:27.389354532 +0000 UTC m=+553.162515055" watchObservedRunningTime="2026-11-01 20:08:27.390139556 +0000 UTC m=+553.163300067" Nov 01 20:08:27.460761 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:27.460677 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6" Nov 01 20:08:27.467555 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:27.467521 5935 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 01 20:08:27.467732 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:27.467583 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6" podUID="86e63000-fc7e-4429-af13-4046198bca02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 01 20:08:28.458318 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:28.449617 5935 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 01 20:08:28.458318 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:28.449684 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6" podUID="86e63000-fc7e-4429-af13-4046198bca02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 01 20:08:29.449316 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:29.448602 5935 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 01 20:08:29.449316 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:29.448660 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6" podUID="86e63000-fc7e-4429-af13-4046198bca02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 01 20:08:30.463676 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:30.463640 5935 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 01 20:08:30.464240 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:30.464210 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6" podUID="86e63000-fc7e-4429-af13-4046198bca02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 01 20:08:31.448665 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:31.448630 5935 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 01 20:08:31.448944 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:31.448914 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6" podUID="86e63000-fc7e-4429-af13-4046198bca02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 01 20:08:32.467737 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:32.467697 5935 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Nov 01 20:08:32.468179 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:32.467780 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6" podUID="86e63000-fc7e-4429-af13-4046198bca02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Nov 01 20:08:33.450637 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:33.450605 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6" Nov 01 20:08:33.450997 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:33.450977 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6" Nov 01 20:08:33.451591 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:33.451572 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6" Nov 01 20:08:35.860101 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:08:35.860057 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:08:38.567354 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:38.567319 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7417-4a3db9"] Nov 01 20:08:38.571066 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:38.571049 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7417-4a3db9" Nov 01 20:08:38.575427 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:38.575406 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-shd7j\"" Nov 01 20:08:38.576771 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:38.576749 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7417-4a3db9"] Nov 01 20:08:38.750192 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:38.750149 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zpz5\" (UniqueName: \"kubernetes.io/projected/127fb1a5-82dd-44d2-8b80-aca91a47b2b9-kube-api-access-7zpz5\") pod \"test-curl-7417-4a3db9\" (UID: \"127fb1a5-82dd-44d2-8b80-aca91a47b2b9\") " pod="opendatahub/test-curl-7417-4a3db9" Nov 01 20:08:38.850904 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:38.850834 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zpz5\" (UniqueName: \"kubernetes.io/projected/127fb1a5-82dd-44d2-8b80-aca91a47b2b9-kube-api-access-7zpz5\") pod \"test-curl-7417-4a3db9\" (UID: \"127fb1a5-82dd-44d2-8b80-aca91a47b2b9\") " pod="opendatahub/test-curl-7417-4a3db9" Nov 01 20:08:38.860482 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:38.860452 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zpz5\" (UniqueName: \"kubernetes.io/projected/127fb1a5-82dd-44d2-8b80-aca91a47b2b9-kube-api-access-7zpz5\") pod \"test-curl-7417-4a3db9\" (UID: \"127fb1a5-82dd-44d2-8b80-aca91a47b2b9\") " pod="opendatahub/test-curl-7417-4a3db9" Nov 01 20:08:38.885596 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:38.885574 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7417-4a3db9" Nov 01 20:08:39.155262 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:39.155241 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7417-4a3db9"] Nov 01 20:08:39.157546 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:08:39.157513 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod127fb1a5_82dd_44d2_8b80_aca91a47b2b9.slice/crio-c8cd3cef0b828710abd4a82ad577a899d65062b470650a7d1c5bb3bfc12a9df2 WatchSource:0}: Error finding container c8cd3cef0b828710abd4a82ad577a899d65062b470650a7d1c5bb3bfc12a9df2: Status 404 returned error can't find the container with id c8cd3cef0b828710abd4a82ad577a899d65062b470650a7d1c5bb3bfc12a9df2 Nov 01 20:08:39.436897 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:39.436821 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7417-4a3db9" event={"ID":"127fb1a5-82dd-44d2-8b80-aca91a47b2b9","Type":"ContainerStarted","Data":"c8cd3cef0b828710abd4a82ad577a899d65062b470650a7d1c5bb3bfc12a9df2"} Nov 01 20:08:40.938531 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:40.938497 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w"] Nov 01 20:08:40.947598 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:40.947576 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w" Nov 01 20:08:40.956373 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:40.953828 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-0aa61b-openshift-default-dockercfg-bv7rw\"" Nov 01 20:08:40.957459 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:40.956854 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w"] Nov 01 20:08:40.976073 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:40.976043 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3118450b-b033-4def-9ccd-39547f6602f4-workload-socket\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w\" (UID: \"3118450b-b033-4def-9ccd-39547f6602f4\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w" Nov 01 20:08:40.976232 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:40.976097 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3118450b-b033-4def-9ccd-39547f6602f4-credential-socket\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w\" (UID: \"3118450b-b033-4def-9ccd-39547f6602f4\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w" Nov 01 20:08:40.976232 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:40.976117 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3118450b-b033-4def-9ccd-39547f6602f4-workload-certs\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w\" (UID: \"3118450b-b033-4def-9ccd-39547f6602f4\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w" Nov 01 20:08:40.976232 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:40.976142 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3118450b-b033-4def-9ccd-39547f6602f4-istio-data\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w\" (UID: \"3118450b-b033-4def-9ccd-39547f6602f4\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w" Nov 01 20:08:40.976232 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:40.976181 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3118450b-b033-4def-9ccd-39547f6602f4-istio-envoy\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w\" (UID: \"3118450b-b033-4def-9ccd-39547f6602f4\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w" Nov 01 20:08:40.976232 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:40.976209 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3118450b-b033-4def-9ccd-39547f6602f4-istio-podinfo\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w\" (UID: \"3118450b-b033-4def-9ccd-39547f6602f4\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w" Nov 01 20:08:40.976232 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:40.976225 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3118450b-b033-4def-9ccd-39547f6602f4-istiod-ca-cert\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w\" (UID: \"3118450b-b033-4def-9ccd-39547f6602f4\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w" Nov 01 20:08:40.976592 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:40.976316 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xp9x\" (UniqueName: \"kubernetes.io/projected/3118450b-b033-4def-9ccd-39547f6602f4-kube-api-access-4xp9x\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w\" (UID: \"3118450b-b033-4def-9ccd-39547f6602f4\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w" Nov 01 20:08:40.976592 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:40.976334 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3118450b-b033-4def-9ccd-39547f6602f4-istio-token\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w\" (UID: \"3118450b-b033-4def-9ccd-39547f6602f4\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w" Nov 01 20:08:41.077079 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:41.077045 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3118450b-b033-4def-9ccd-39547f6602f4-istio-podinfo\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w\" (UID: \"3118450b-b033-4def-9ccd-39547f6602f4\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w" Nov 01 20:08:41.077272 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:41.077085 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3118450b-b033-4def-9ccd-39547f6602f4-istiod-ca-cert\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w\" (UID: \"3118450b-b033-4def-9ccd-39547f6602f4\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w" Nov 01 20:08:41.077272 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:41.077156 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xp9x\" (UniqueName: \"kubernetes.io/projected/3118450b-b033-4def-9ccd-39547f6602f4-kube-api-access-4xp9x\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w\" (UID: \"3118450b-b033-4def-9ccd-39547f6602f4\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w" Nov 01 20:08:41.077272 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:41.077182 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3118450b-b033-4def-9ccd-39547f6602f4-istio-token\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w\" (UID: \"3118450b-b033-4def-9ccd-39547f6602f4\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w" Nov 01 20:08:41.077272 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:41.077222 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3118450b-b033-4def-9ccd-39547f6602f4-workload-socket\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w\" (UID: \"3118450b-b033-4def-9ccd-39547f6602f4\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w" Nov 01 20:08:41.077541 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:41.077268 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3118450b-b033-4def-9ccd-39547f6602f4-credential-socket\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w\" (UID: \"3118450b-b033-4def-9ccd-39547f6602f4\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w" Nov 01 20:08:41.077541 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:41.077324 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3118450b-b033-4def-9ccd-39547f6602f4-workload-certs\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w\" (UID: \"3118450b-b033-4def-9ccd-39547f6602f4\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w" Nov 01 20:08:41.077541 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:41.077368 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3118450b-b033-4def-9ccd-39547f6602f4-istio-data\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w\" (UID: \"3118450b-b033-4def-9ccd-39547f6602f4\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w" Nov 01 20:08:41.077541 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:41.077425 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3118450b-b033-4def-9ccd-39547f6602f4-istio-envoy\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w\" (UID: \"3118450b-b033-4def-9ccd-39547f6602f4\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w" Nov 01 20:08:41.077941 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:41.077865 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3118450b-b033-4def-9ccd-39547f6602f4-credential-socket\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w\" (UID: \"3118450b-b033-4def-9ccd-39547f6602f4\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w" Nov 01 20:08:41.078070 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:41.077953 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3118450b-b033-4def-9ccd-39547f6602f4-istiod-ca-cert\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w\" (UID: \"3118450b-b033-4def-9ccd-39547f6602f4\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w" Nov 01 20:08:41.078070 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:41.077955 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3118450b-b033-4def-9ccd-39547f6602f4-workload-certs\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w\" (UID: \"3118450b-b033-4def-9ccd-39547f6602f4\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w" Nov 01 20:08:41.078190 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:41.078116 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3118450b-b033-4def-9ccd-39547f6602f4-istio-data\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w\" (UID: \"3118450b-b033-4def-9ccd-39547f6602f4\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w" Nov 01 20:08:41.078581 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:41.078555 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3118450b-b033-4def-9ccd-39547f6602f4-workload-socket\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w\" (UID: \"3118450b-b033-4def-9ccd-39547f6602f4\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w" Nov 01 20:08:41.080632 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:41.080608 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3118450b-b033-4def-9ccd-39547f6602f4-istio-envoy\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w\" (UID: \"3118450b-b033-4def-9ccd-39547f6602f4\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w" Nov 01 20:08:41.080944 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:41.080916 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3118450b-b033-4def-9ccd-39547f6602f4-istio-podinfo\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w\" (UID: \"3118450b-b033-4def-9ccd-39547f6602f4\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w" Nov 01 20:08:41.085480 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:41.085455 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3118450b-b033-4def-9ccd-39547f6602f4-istio-token\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w\" (UID: \"3118450b-b033-4def-9ccd-39547f6602f4\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w" Nov 01 20:08:41.085579 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:41.085512 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xp9x\" (UniqueName: \"kubernetes.io/projected/3118450b-b033-4def-9ccd-39547f6602f4-kube-api-access-4xp9x\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w\" (UID: \"3118450b-b033-4def-9ccd-39547f6602f4\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w" Nov 01 20:08:41.266228 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:41.266196 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w" Nov 01 20:08:41.568968 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:41.568945 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w"] Nov 01 20:08:41.570389 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:08:41.570360 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3118450b_b033_4def_9ccd_39547f6602f4.slice/crio-d45505a6e16292a4e669e905c74bdf5a2cbe9787e99a62b59721d5cbeddd50d1 WatchSource:0}: Error finding container d45505a6e16292a4e669e905c74bdf5a2cbe9787e99a62b59721d5cbeddd50d1: Status 404 returned error can't find the container with id d45505a6e16292a4e669e905c74bdf5a2cbe9787e99a62b59721d5cbeddd50d1 Nov 01 20:08:41.572871 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:41.572835 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:08:41.572959 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:41.572915 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:08:41.573007 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:41.572957 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:08:42.473086 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:42.473047 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7417-4a3db9" event={"ID":"127fb1a5-82dd-44d2-8b80-aca91a47b2b9","Type":"ContainerStarted","Data":"83dbfcad47c43dfcd9a21620cab6bfe431eb809d385d9a027a05fbade4ed3955"} Nov 01 20:08:42.474593 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:42.474571 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w" event={"ID":"3118450b-b033-4def-9ccd-39547f6602f4","Type":"ContainerStarted","Data":"2a8ea06c755e317176ba32e3fd0bb4c75fd4414b205f92f1642cad3f0d618348"} Nov 01 20:08:42.474658 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:42.474614 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w" event={"ID":"3118450b-b033-4def-9ccd-39547f6602f4","Type":"ContainerStarted","Data":"d45505a6e16292a4e669e905c74bdf5a2cbe9787e99a62b59721d5cbeddd50d1"} Nov 01 20:08:42.487447 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:42.487411 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7417-4a3db9" podStartSLOduration=2.116705218 podStartE2EDuration="4.487400816s" totalImagesPullingTime="2.370695598s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:08:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:08:42.486464071 +0000 UTC m=+568.259624583" watchObservedRunningTime="2026-11-01 20:08:42.487400816 +0000 UTC m=+568.260561327" Nov 01 20:08:42.504424 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:42.504372 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w" podStartSLOduration=2.504357656 podStartE2EDuration="2.504357656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:08:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:08:42.502983269 +0000 UTC m=+568.276143809" watchObservedRunningTime="2026-11-01 20:08:42.504357656 +0000 UTC m=+568.277518169" Nov 01 20:08:43.117222 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:43.117183 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7417-4a3db9"] Nov 01 20:08:43.121104 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:43.121075 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7417-4a3db9"] Nov 01 20:08:43.267662 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:43.267551 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w" Nov 01 20:08:43.268803 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:43.268775 5935 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Nov 01 20:08:43.268955 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:43.268817 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w" podUID="3118450b-b033-4def-9ccd-39547f6602f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Nov 01 20:08:43.273962 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:43.273942 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7417-a77bc6"] Nov 01 20:08:43.274484 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:43.274470 5935 cpu_manager.go:414] "containerMap: removing container" podUID="127fb1a5-82dd-44d2-8b80-aca91a47b2b9" containerName="test-curl-7417-4a3db9" Nov 01 20:08:43.274524 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:43.274486 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="127fb1a5-82dd-44d2-8b80-aca91a47b2b9" containerName="test-curl-7417-4a3db9" Nov 01 20:08:43.274593 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:43.274584 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="127fb1a5-82dd-44d2-8b80-aca91a47b2b9" containerName="test-curl-7417-4a3db9" Nov 01 20:08:43.278412 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:43.278397 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7417-a77bc6" Nov 01 20:08:43.284023 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:43.283331 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7417-a77bc6"] Nov 01 20:08:43.300190 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:43.300167 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gc9v\" (UniqueName: \"kubernetes.io/projected/0b3e60e2-b842-4095-a97a-efab9733867c-kube-api-access-6gc9v\") pod \"test-curl-7417-a77bc6\" (UID: \"0b3e60e2-b842-4095-a97a-efab9733867c\") " pod="opendatahub/test-curl-7417-a77bc6" Nov 01 20:08:43.401309 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:43.401188 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gc9v\" (UniqueName: \"kubernetes.io/projected/0b3e60e2-b842-4095-a97a-efab9733867c-kube-api-access-6gc9v\") pod \"test-curl-7417-a77bc6\" (UID: \"0b3e60e2-b842-4095-a97a-efab9733867c\") " pod="opendatahub/test-curl-7417-a77bc6" Nov 01 20:08:43.409222 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:43.409198 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gc9v\" (UniqueName: \"kubernetes.io/projected/0b3e60e2-b842-4095-a97a-efab9733867c-kube-api-access-6gc9v\") pod \"test-curl-7417-a77bc6\" (UID: \"0b3e60e2-b842-4095-a97a-efab9733867c\") " pod="opendatahub/test-curl-7417-a77bc6" Nov 01 20:08:43.592167 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:43.592139 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7417-a77bc6" Nov 01 20:08:43.892940 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:43.892912 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7417-a77bc6"] Nov 01 20:08:43.894530 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:08:43.894506 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b3e60e2_b842_4095_a97a_efab9733867c.slice/crio-dcf714fe8c7ceea2f70fe904ddebffd6807e82b80c5b9da1121d087e7dbea162 WatchSource:0}: Error finding container dcf714fe8c7ceea2f70fe904ddebffd6807e82b80c5b9da1121d087e7dbea162: Status 404 returned error can't find the container with id dcf714fe8c7ceea2f70fe904ddebffd6807e82b80c5b9da1121d087e7dbea162 Nov 01 20:08:44.267691 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:44.267643 5935 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Nov 01 20:08:44.267889 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:44.267702 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w" podUID="3118450b-b033-4def-9ccd-39547f6602f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Nov 01 20:08:44.489804 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:44.489769 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7417-a77bc6" event={"ID":"0b3e60e2-b842-4095-a97a-efab9733867c","Type":"ContainerStarted","Data":"0b4728db4ab9e8d475bc0542f5f62854f57d7d4ffe1b70788c33a83f3954bda6"} Nov 01 20:08:44.490407 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:44.490389 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7417-a77bc6" event={"ID":"0b3e60e2-b842-4095-a97a-efab9733867c","Type":"ContainerStarted","Data":"dcf714fe8c7ceea2f70fe904ddebffd6807e82b80c5b9da1121d087e7dbea162"} Nov 01 20:08:44.490613 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:44.490163 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7417-4a3db9" podUID="127fb1a5-82dd-44d2-8b80-aca91a47b2b9" containerName="test-curl-7417-4a3db9" containerID="cri-o://83dbfcad47c43dfcd9a21620cab6bfe431eb809d385d9a027a05fbade4ed3955" gracePeriod=2 Nov 01 20:08:45.266309 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:45.253860 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7417-a77bc6" podStartSLOduration=2.253841858 podStartE2EDuration="2.253841858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:08:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:08:44.504634662 +0000 UTC m=+570.277795174" watchObservedRunningTime="2026-11-01 20:08:45.253841858 +0000 UTC m=+571.027002367" Nov 01 20:08:45.266309 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:45.255838 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7417-a77bc6"] Nov 01 20:08:45.266309 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:45.259100 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7417-a77bc6"] Nov 01 20:08:45.270308 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:45.269634 5935 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Nov 01 20:08:45.270308 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:45.269683 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w" podUID="3118450b-b033-4def-9ccd-39547f6602f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Nov 01 20:08:46.268264 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:46.268225 5935 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Nov 01 20:08:46.268690 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:46.268305 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w" podUID="3118450b-b033-4def-9ccd-39547f6602f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Nov 01 20:08:46.508344 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:46.504197 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7417-a77bc6" podUID="0b3e60e2-b842-4095-a97a-efab9733867c" containerName="test-curl-7417-a77bc6" containerID="cri-o://0b4728db4ab9e8d475bc0542f5f62854f57d7d4ffe1b70788c33a83f3954bda6" gracePeriod=2 Nov 01 20:08:46.695773 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:46.695748 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7417-4a3db9" Nov 01 20:08:46.739349 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:46.739321 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/127fb1a5-82dd-44d2-8b80-aca91a47b2b9-kube-api-access-7zpz5\" (UniqueName: \"kubernetes.io/projected/127fb1a5-82dd-44d2-8b80-aca91a47b2b9-kube-api-access-7zpz5\") pod \"127fb1a5-82dd-44d2-8b80-aca91a47b2b9\" (UID: \"127fb1a5-82dd-44d2-8b80-aca91a47b2b9\") " Nov 01 20:08:46.741199 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:46.741176 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/127fb1a5-82dd-44d2-8b80-aca91a47b2b9-kube-api-access-7zpz5" pod "127fb1a5-82dd-44d2-8b80-aca91a47b2b9" (UID: "127fb1a5-82dd-44d2-8b80-aca91a47b2b9"). InnerVolumeSpecName "kube-api-access-7zpz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:08:46.840992 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:46.840960 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zpz5\" (UniqueName: \"kubernetes.io/projected/127fb1a5-82dd-44d2-8b80-aca91a47b2b9-kube-api-access-7zpz5\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:08:47.267308 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:47.266713 5935 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Nov 01 20:08:47.267308 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:47.266804 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w" podUID="3118450b-b033-4def-9ccd-39547f6602f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Nov 01 20:08:47.513860 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:47.513822 5935 generic.go:356] "Generic (PLEG): container finished" podID="127fb1a5-82dd-44d2-8b80-aca91a47b2b9" containerID="83dbfcad47c43dfcd9a21620cab6bfe431eb809d385d9a027a05fbade4ed3955" exitCode=137 Nov 01 20:08:47.514243 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:47.513973 5935 scope.go:122] "RemoveContainer" containerID="83dbfcad47c43dfcd9a21620cab6bfe431eb809d385d9a027a05fbade4ed3955" Nov 01 20:08:47.514243 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:47.514133 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7417-4a3db9" Nov 01 20:08:47.527806 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:47.527772 5935 scope.go:122] "RemoveContainer" containerID="83dbfcad47c43dfcd9a21620cab6bfe431eb809d385d9a027a05fbade4ed3955" Nov 01 20:08:47.528159 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:08:47.528121 5935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83dbfcad47c43dfcd9a21620cab6bfe431eb809d385d9a027a05fbade4ed3955\": container with ID starting with 83dbfcad47c43dfcd9a21620cab6bfe431eb809d385d9a027a05fbade4ed3955 not found: ID does not exist" containerID="83dbfcad47c43dfcd9a21620cab6bfe431eb809d385d9a027a05fbade4ed3955" Nov 01 20:08:47.528268 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:47.528157 5935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83dbfcad47c43dfcd9a21620cab6bfe431eb809d385d9a027a05fbade4ed3955"} err="failed to get container status \"83dbfcad47c43dfcd9a21620cab6bfe431eb809d385d9a027a05fbade4ed3955\": rpc error: code = NotFound desc = could not find container \"83dbfcad47c43dfcd9a21620cab6bfe431eb809d385d9a027a05fbade4ed3955\": container with ID starting with 83dbfcad47c43dfcd9a21620cab6bfe431eb809d385d9a027a05fbade4ed3955 not found: ID does not exist" Nov 01 20:08:47.529784 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:08:47.529755 5935 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7417-4a3db9\" is forbidden: User \"system:node:ip-10-0-136-26.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-26.us-east-2.compute.internal' and this object" podUID="127fb1a5-82dd-44d2-8b80-aca91a47b2b9" pod="opendatahub/test-curl-7417-4a3db9" Nov 01 20:08:48.266696 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:48.266666 5935 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Nov 01 20:08:48.266873 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:48.266732 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w" podUID="3118450b-b033-4def-9ccd-39547f6602f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Nov 01 20:08:48.354586 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:48.354554 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="127fb1a5-82dd-44d2-8b80-aca91a47b2b9" path="/var/lib/kubelet/pods/127fb1a5-82dd-44d2-8b80-aca91a47b2b9/volumes" Nov 01 20:08:48.720793 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:48.720766 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7417-a77bc6" Nov 01 20:08:48.798645 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:48.798586 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b3e60e2-b842-4095-a97a-efab9733867c-kube-api-access-6gc9v\" (UniqueName: \"kubernetes.io/projected/0b3e60e2-b842-4095-a97a-efab9733867c-kube-api-access-6gc9v\") pod \"0b3e60e2-b842-4095-a97a-efab9733867c\" (UID: \"0b3e60e2-b842-4095-a97a-efab9733867c\") " Nov 01 20:08:48.800735 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:48.800707 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b3e60e2-b842-4095-a97a-efab9733867c-kube-api-access-6gc9v" pod "0b3e60e2-b842-4095-a97a-efab9733867c" (UID: "0b3e60e2-b842-4095-a97a-efab9733867c"). InnerVolumeSpecName "kube-api-access-6gc9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:08:48.900111 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:48.900074 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gc9v\" (UniqueName: \"kubernetes.io/projected/0b3e60e2-b842-4095-a97a-efab9733867c-kube-api-access-6gc9v\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:08:49.526768 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:49.526729 5935 generic.go:356] "Generic (PLEG): container finished" podID="0b3e60e2-b842-4095-a97a-efab9733867c" containerID="0b4728db4ab9e8d475bc0542f5f62854f57d7d4ffe1b70788c33a83f3954bda6" exitCode=137 Nov 01 20:08:49.526768 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:49.526773 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7417-a77bc6" Nov 01 20:08:49.527021 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:49.526831 5935 scope.go:122] "RemoveContainer" containerID="0b4728db4ab9e8d475bc0542f5f62854f57d7d4ffe1b70788c33a83f3954bda6" Nov 01 20:08:49.536514 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:49.536496 5935 scope.go:122] "RemoveContainer" containerID="0b4728db4ab9e8d475bc0542f5f62854f57d7d4ffe1b70788c33a83f3954bda6" Nov 01 20:08:49.536766 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:08:49.536747 5935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b4728db4ab9e8d475bc0542f5f62854f57d7d4ffe1b70788c33a83f3954bda6\": container with ID starting with 0b4728db4ab9e8d475bc0542f5f62854f57d7d4ffe1b70788c33a83f3954bda6 not found: ID does not exist" containerID="0b4728db4ab9e8d475bc0542f5f62854f57d7d4ffe1b70788c33a83f3954bda6" Nov 01 20:08:49.536835 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:49.536772 5935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b4728db4ab9e8d475bc0542f5f62854f57d7d4ffe1b70788c33a83f3954bda6"} err="failed to get container status \"0b4728db4ab9e8d475bc0542f5f62854f57d7d4ffe1b70788c33a83f3954bda6\": rpc error: code = NotFound desc = could not find container \"0b4728db4ab9e8d475bc0542f5f62854f57d7d4ffe1b70788c33a83f3954bda6\": container with ID starting with 0b4728db4ab9e8d475bc0542f5f62854f57d7d4ffe1b70788c33a83f3954bda6 not found: ID does not exist" Nov 01 20:08:49.540527 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:08:49.540502 5935 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7417-a77bc6\" is forbidden: User \"system:node:ip-10-0-136-26.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-26.us-east-2.compute.internal' and this object" podUID="0b3e60e2-b842-4095-a97a-efab9733867c" pod="opendatahub/test-curl-7417-a77bc6" Nov 01 20:08:49.950769 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:49.950740 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w" Nov 01 20:08:49.951266 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:49.951244 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w" Nov 01 20:08:49.952615 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:49.952596 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w" Nov 01 20:08:49.952851 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:08:49.952830 5935 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7417-a77bc6\" is forbidden: User \"system:node:ip-10-0-136-26.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-26.us-east-2.compute.internal' and this object" podUID="0b3e60e2-b842-4095-a97a-efab9733867c" pod="opendatahub/test-curl-7417-a77bc6" Nov 01 20:08:49.990695 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:08:49.990660 5935 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7417-a77bc6\" is forbidden: User \"system:node:ip-10-0-136-26.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-26.us-east-2.compute.internal' and this object" podUID="0b3e60e2-b842-4095-a97a-efab9733867c" pod="opendatahub/test-curl-7417-a77bc6" Nov 01 20:08:50.352474 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:50.352379 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b3e60e2-b842-4095-a97a-efab9733867c" path="/var/lib/kubelet/pods/0b3e60e2-b842-4095-a97a-efab9733867c/volumes" Nov 01 20:08:55.292962 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:55.292911 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-e38ef4-79f5864c9d-xbg7r"] Nov 01 20:08:55.293636 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:55.293614 5935 cpu_manager.go:414] "containerMap: removing container" podUID="0b3e60e2-b842-4095-a97a-efab9733867c" containerName="test-curl-7417-a77bc6" Nov 01 20:08:55.293678 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:55.293640 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b3e60e2-b842-4095-a97a-efab9733867c" containerName="test-curl-7417-a77bc6" Nov 01 20:08:55.293826 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:55.293800 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b3e60e2-b842-4095-a97a-efab9733867c" containerName="test-curl-7417-a77bc6" Nov 01 20:08:55.319232 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:55.319205 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e38ef4-79f5864c9d-xbg7r" Nov 01 20:08:55.323691 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:55.323665 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-e38ef4-79f5864c9d-xbg7r"] Nov 01 20:08:55.327555 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:55.327520 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-e38ef4-serving-cert\"" Nov 01 20:08:55.467077 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:55.467033 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1238788e-4bf8-4a19-8de9-809061a7cb2b-metrics-certs\") pod \"maas-api-e2e-extmod-a-e38ef4-79f5864c9d-xbg7r\" (UID: \"1238788e-4bf8-4a19-8de9-809061a7cb2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e38ef4-79f5864c9d-xbg7r" Nov 01 20:08:55.467281 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:55.467102 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j99ds\" (UniqueName: \"kubernetes.io/projected/1238788e-4bf8-4a19-8de9-809061a7cb2b-kube-api-access-j99ds\") pod \"maas-api-e2e-extmod-a-e38ef4-79f5864c9d-xbg7r\" (UID: \"1238788e-4bf8-4a19-8de9-809061a7cb2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e38ef4-79f5864c9d-xbg7r" Nov 01 20:08:55.467281 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:55.467257 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1238788e-4bf8-4a19-8de9-809061a7cb2b-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-e38ef4-79f5864c9d-xbg7r\" (UID: \"1238788e-4bf8-4a19-8de9-809061a7cb2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e38ef4-79f5864c9d-xbg7r" Nov 01 20:08:55.467428 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:55.467374 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1238788e-4bf8-4a19-8de9-809061a7cb2b-maas-api-tls\") pod \"maas-api-e2e-extmod-a-e38ef4-79f5864c9d-xbg7r\" (UID: \"1238788e-4bf8-4a19-8de9-809061a7cb2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e38ef4-79f5864c9d-xbg7r" Nov 01 20:08:55.568490 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:55.568407 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1238788e-4bf8-4a19-8de9-809061a7cb2b-maas-api-tls\") pod \"maas-api-e2e-extmod-a-e38ef4-79f5864c9d-xbg7r\" (UID: \"1238788e-4bf8-4a19-8de9-809061a7cb2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e38ef4-79f5864c9d-xbg7r" Nov 01 20:08:55.568644 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:55.568546 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1238788e-4bf8-4a19-8de9-809061a7cb2b-metrics-certs\") pod \"maas-api-e2e-extmod-a-e38ef4-79f5864c9d-xbg7r\" (UID: \"1238788e-4bf8-4a19-8de9-809061a7cb2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e38ef4-79f5864c9d-xbg7r" Nov 01 20:08:55.568644 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:55.568590 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j99ds\" (UniqueName: \"kubernetes.io/projected/1238788e-4bf8-4a19-8de9-809061a7cb2b-kube-api-access-j99ds\") pod \"maas-api-e2e-extmod-a-e38ef4-79f5864c9d-xbg7r\" (UID: \"1238788e-4bf8-4a19-8de9-809061a7cb2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e38ef4-79f5864c9d-xbg7r" Nov 01 20:08:55.568644 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:55.568617 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1238788e-4bf8-4a19-8de9-809061a7cb2b-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-e38ef4-79f5864c9d-xbg7r\" (UID: \"1238788e-4bf8-4a19-8de9-809061a7cb2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e38ef4-79f5864c9d-xbg7r" Nov 01 20:08:55.569381 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:55.569356 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1238788e-4bf8-4a19-8de9-809061a7cb2b-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-e38ef4-79f5864c9d-xbg7r\" (UID: \"1238788e-4bf8-4a19-8de9-809061a7cb2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e38ef4-79f5864c9d-xbg7r" Nov 01 20:08:55.571082 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:55.571055 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1238788e-4bf8-4a19-8de9-809061a7cb2b-metrics-certs\") pod \"maas-api-e2e-extmod-a-e38ef4-79f5864c9d-xbg7r\" (UID: \"1238788e-4bf8-4a19-8de9-809061a7cb2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e38ef4-79f5864c9d-xbg7r" Nov 01 20:08:55.571460 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:55.571432 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1238788e-4bf8-4a19-8de9-809061a7cb2b-maas-api-tls\") pod \"maas-api-e2e-extmod-a-e38ef4-79f5864c9d-xbg7r\" (UID: \"1238788e-4bf8-4a19-8de9-809061a7cb2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e38ef4-79f5864c9d-xbg7r" Nov 01 20:08:55.576631 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:55.576607 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j99ds\" (UniqueName: \"kubernetes.io/projected/1238788e-4bf8-4a19-8de9-809061a7cb2b-kube-api-access-j99ds\") pod \"maas-api-e2e-extmod-a-e38ef4-79f5864c9d-xbg7r\" (UID: \"1238788e-4bf8-4a19-8de9-809061a7cb2b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e38ef4-79f5864c9d-xbg7r" Nov 01 20:08:55.640090 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:55.640058 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e38ef4-79f5864c9d-xbg7r" Nov 01 20:08:55.944543 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:55.944126 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-e38ef4-79f5864c9d-xbg7r"] Nov 01 20:08:55.951625 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:08:55.951593 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1238788e_4bf8_4a19_8de9_809061a7cb2b.slice/crio-cd4b77b6616136f830ec753da3bbeaae75699233bf28c82e17a30a34e9ddfb90 WatchSource:0}: Error finding container cd4b77b6616136f830ec753da3bbeaae75699233bf28c82e17a30a34e9ddfb90: Status 404 returned error can't find the container with id cd4b77b6616136f830ec753da3bbeaae75699233bf28c82e17a30a34e9ddfb90 Nov 01 20:08:56.169139 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:56.169109 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb"] Nov 01 20:08:56.174092 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:56.174071 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb" Nov 01 20:08:56.186358 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:56.186326 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb"] Nov 01 20:08:56.281101 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:56.281027 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8249130-8b25-468a-9904-2564d38fdb15-istio-podinfo\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb\" (UID: \"f8249130-8b25-468a-9904-2564d38fdb15\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb" Nov 01 20:08:56.281101 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:56.281082 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8249130-8b25-468a-9904-2564d38fdb15-istio-envoy\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb\" (UID: \"f8249130-8b25-468a-9904-2564d38fdb15\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb" Nov 01 20:08:56.281277 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:56.281167 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8249130-8b25-468a-9904-2564d38fdb15-istio-data\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb\" (UID: \"f8249130-8b25-468a-9904-2564d38fdb15\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb" Nov 01 20:08:56.281277 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:56.281191 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8249130-8b25-468a-9904-2564d38fdb15-workload-certs\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb\" (UID: \"f8249130-8b25-468a-9904-2564d38fdb15\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb" Nov 01 20:08:56.281383 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:56.281306 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8249130-8b25-468a-9904-2564d38fdb15-istiod-ca-cert\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb\" (UID: \"f8249130-8b25-468a-9904-2564d38fdb15\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb" Nov 01 20:08:56.281383 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:56.281366 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vltmj\" (UniqueName: \"kubernetes.io/projected/f8249130-8b25-468a-9904-2564d38fdb15-kube-api-access-vltmj\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb\" (UID: \"f8249130-8b25-468a-9904-2564d38fdb15\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb" Nov 01 20:08:56.281460 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:56.281420 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8249130-8b25-468a-9904-2564d38fdb15-credential-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb\" (UID: \"f8249130-8b25-468a-9904-2564d38fdb15\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb" Nov 01 20:08:56.281500 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:56.281487 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8249130-8b25-468a-9904-2564d38fdb15-istio-token\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb\" (UID: \"f8249130-8b25-468a-9904-2564d38fdb15\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb" Nov 01 20:08:56.281533 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:56.281511 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8249130-8b25-468a-9904-2564d38fdb15-workload-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb\" (UID: \"f8249130-8b25-468a-9904-2564d38fdb15\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb" Nov 01 20:08:56.382179 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:56.382140 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8249130-8b25-468a-9904-2564d38fdb15-istio-envoy\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb\" (UID: \"f8249130-8b25-468a-9904-2564d38fdb15\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb" Nov 01 20:08:56.382740 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:56.382196 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8249130-8b25-468a-9904-2564d38fdb15-istio-data\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb\" (UID: \"f8249130-8b25-468a-9904-2564d38fdb15\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb" Nov 01 20:08:56.382740 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:56.382231 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8249130-8b25-468a-9904-2564d38fdb15-workload-certs\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb\" (UID: \"f8249130-8b25-468a-9904-2564d38fdb15\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb" Nov 01 20:08:56.382740 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:56.382338 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8249130-8b25-468a-9904-2564d38fdb15-istiod-ca-cert\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb\" (UID: \"f8249130-8b25-468a-9904-2564d38fdb15\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb" Nov 01 20:08:56.382740 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:56.382411 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vltmj\" (UniqueName: \"kubernetes.io/projected/f8249130-8b25-468a-9904-2564d38fdb15-kube-api-access-vltmj\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb\" (UID: \"f8249130-8b25-468a-9904-2564d38fdb15\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb" Nov 01 20:08:56.382740 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:56.382459 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8249130-8b25-468a-9904-2564d38fdb15-credential-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb\" (UID: \"f8249130-8b25-468a-9904-2564d38fdb15\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb" Nov 01 20:08:56.382740 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:56.382503 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8249130-8b25-468a-9904-2564d38fdb15-istio-token\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb\" (UID: \"f8249130-8b25-468a-9904-2564d38fdb15\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb" Nov 01 20:08:56.382740 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:56.382533 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8249130-8b25-468a-9904-2564d38fdb15-workload-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb\" (UID: \"f8249130-8b25-468a-9904-2564d38fdb15\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb" Nov 01 20:08:56.382740 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:56.382567 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8249130-8b25-468a-9904-2564d38fdb15-istio-podinfo\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb\" (UID: \"f8249130-8b25-468a-9904-2564d38fdb15\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb" Nov 01 20:08:56.383359 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:56.383323 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8249130-8b25-468a-9904-2564d38fdb15-istiod-ca-cert\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb\" (UID: \"f8249130-8b25-468a-9904-2564d38fdb15\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb" Nov 01 20:08:56.383608 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:56.383585 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8249130-8b25-468a-9904-2564d38fdb15-credential-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb\" (UID: \"f8249130-8b25-468a-9904-2564d38fdb15\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb" Nov 01 20:08:56.383815 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:56.383798 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8249130-8b25-468a-9904-2564d38fdb15-workload-certs\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb\" (UID: \"f8249130-8b25-468a-9904-2564d38fdb15\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb" Nov 01 20:08:56.384646 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:56.384621 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8249130-8b25-468a-9904-2564d38fdb15-workload-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb\" (UID: \"f8249130-8b25-468a-9904-2564d38fdb15\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb" Nov 01 20:08:56.385268 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:56.385245 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8249130-8b25-468a-9904-2564d38fdb15-istio-data\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb\" (UID: \"f8249130-8b25-468a-9904-2564d38fdb15\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb" Nov 01 20:08:56.385453 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:56.385432 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8249130-8b25-468a-9904-2564d38fdb15-istio-podinfo\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb\" (UID: \"f8249130-8b25-468a-9904-2564d38fdb15\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb" Nov 01 20:08:56.386709 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:56.386687 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8249130-8b25-468a-9904-2564d38fdb15-istio-envoy\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb\" (UID: \"f8249130-8b25-468a-9904-2564d38fdb15\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb" Nov 01 20:08:56.392584 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:56.392557 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8249130-8b25-468a-9904-2564d38fdb15-istio-token\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb\" (UID: \"f8249130-8b25-468a-9904-2564d38fdb15\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb" Nov 01 20:08:56.393034 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:56.393013 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vltmj\" (UniqueName: \"kubernetes.io/projected/f8249130-8b25-468a-9904-2564d38fdb15-kube-api-access-vltmj\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb\" (UID: \"f8249130-8b25-468a-9904-2564d38fdb15\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb" Nov 01 20:08:56.492238 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:56.492214 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb" Nov 01 20:08:56.571515 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:56.571219 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e38ef4-79f5864c9d-xbg7r" event={"ID":"1238788e-4bf8-4a19-8de9-809061a7cb2b","Type":"ContainerStarted","Data":"7f888c5280dd1c9f5dd21cc9dd7ab772c23336f638e3ae0fcc8ab92f4dd317ce"} Nov 01 20:08:56.571515 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:56.571298 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e38ef4-79f5864c9d-xbg7r" event={"ID":"1238788e-4bf8-4a19-8de9-809061a7cb2b","Type":"ContainerStarted","Data":"cd4b77b6616136f830ec753da3bbeaae75699233bf28c82e17a30a34e9ddfb90"} Nov 01 20:08:56.572257 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:56.572233 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e38ef4-79f5864c9d-xbg7r" Nov 01 20:08:56.595099 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:56.595044 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e38ef4-79f5864c9d-xbg7r" podStartSLOduration=1.072542145 podStartE2EDuration="1.595025931s" totalImagesPullingTime="522.483786ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:08:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:08:56.590408737 +0000 UTC m=+582.363569249" watchObservedRunningTime="2026-11-01 20:08:56.595025931 +0000 UTC m=+582.368186443" Nov 01 20:08:56.778158 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:56.778132 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb"] Nov 01 20:08:56.779937 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:08:56.779909 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8249130_8b25_468a_9904_2564d38fdb15.slice/crio-7e2423d2117f2f7abeed4a6c8529c6a1f47cdd52d6563aead0e6c5dc921b216c WatchSource:0}: Error finding container 7e2423d2117f2f7abeed4a6c8529c6a1f47cdd52d6563aead0e6c5dc921b216c: Status 404 returned error can't find the container with id 7e2423d2117f2f7abeed4a6c8529c6a1f47cdd52d6563aead0e6c5dc921b216c Nov 01 20:08:56.783266 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:56.783193 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:08:56.783377 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:56.783298 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:08:56.783377 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:56.783352 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:08:57.578525 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:57.578487 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb" event={"ID":"f8249130-8b25-468a-9904-2564d38fdb15","Type":"ContainerStarted","Data":"c74aa07d56563c2668fc0af4c188dabcff2d8a9a247f4dd9ec20736493faac6a"} Nov 01 20:08:57.578922 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:57.578547 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb" event={"ID":"f8249130-8b25-468a-9904-2564d38fdb15","Type":"ContainerStarted","Data":"7e2423d2117f2f7abeed4a6c8529c6a1f47cdd52d6563aead0e6c5dc921b216c"} Nov 01 20:08:57.601842 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:57.601790 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb" podStartSLOduration=1.6017773 podStartE2EDuration="1.6017773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:08:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:08:57.599097433 +0000 UTC m=+583.372257980" watchObservedRunningTime="2026-11-01 20:08:57.6017773 +0000 UTC m=+583.374937812" Nov 01 20:08:58.492616 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:58.492581 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb" Nov 01 20:08:58.494060 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:58.494037 5935 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Nov 01 20:08:58.494168 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:58.494083 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb" podUID="f8249130-8b25-468a-9904-2564d38fdb15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Nov 01 20:08:59.492891 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:59.492861 5935 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Nov 01 20:08:59.493296 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:08:59.492914 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb" podUID="f8249130-8b25-468a-9904-2564d38fdb15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Nov 01 20:09:00.493058 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:00.493010 5935 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Nov 01 20:09:00.493545 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:00.493085 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb" podUID="f8249130-8b25-468a-9904-2564d38fdb15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Nov 01 20:09:01.492883 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:01.492851 5935 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Nov 01 20:09:01.493059 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:01.492902 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb" podUID="f8249130-8b25-468a-9904-2564d38fdb15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Nov 01 20:09:02.492904 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:02.492866 5935 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Nov 01 20:09:02.493427 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:02.492939 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb" podUID="f8249130-8b25-468a-9904-2564d38fdb15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Nov 01 20:09:03.589411 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:03.589382 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e38ef4-79f5864c9d-xbg7r" Nov 01 20:09:03.647671 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:03.647640 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb" Nov 01 20:09:03.648518 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:03.648498 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb" Nov 01 20:09:03.649451 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:03.649431 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb" Nov 01 20:09:05.909871 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:09:05.909827 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:09:11.109275 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:11.109239 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp"] Nov 01 20:09:11.113544 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:11.113522 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp" Nov 01 20:09:11.123419 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:11.123398 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp"] Nov 01 20:09:11.191664 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:11.191638 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2b197fc-6182-4f88-8e11-d631a9d95b37-istio-envoy\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp\" (UID: \"e2b197fc-6182-4f88-8e11-d631a9d95b37\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp" Nov 01 20:09:11.191816 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:11.191721 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2b197fc-6182-4f88-8e11-d631a9d95b37-workload-socket\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp\" (UID: \"e2b197fc-6182-4f88-8e11-d631a9d95b37\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp" Nov 01 20:09:11.191816 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:11.191741 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2b197fc-6182-4f88-8e11-d631a9d95b37-istio-data\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp\" (UID: \"e2b197fc-6182-4f88-8e11-d631a9d95b37\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp" Nov 01 20:09:11.191816 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:11.191763 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2b197fc-6182-4f88-8e11-d631a9d95b37-istio-token\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp\" (UID: \"e2b197fc-6182-4f88-8e11-d631a9d95b37\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp" Nov 01 20:09:11.191816 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:11.191803 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2b197fc-6182-4f88-8e11-d631a9d95b37-credential-socket\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp\" (UID: \"e2b197fc-6182-4f88-8e11-d631a9d95b37\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp" Nov 01 20:09:11.191947 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:11.191884 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhnzz\" (UniqueName: \"kubernetes.io/projected/e2b197fc-6182-4f88-8e11-d631a9d95b37-kube-api-access-nhnzz\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp\" (UID: \"e2b197fc-6182-4f88-8e11-d631a9d95b37\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp" Nov 01 20:09:11.191992 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:11.191978 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2b197fc-6182-4f88-8e11-d631a9d95b37-istio-podinfo\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp\" (UID: \"e2b197fc-6182-4f88-8e11-d631a9d95b37\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp" Nov 01 20:09:11.192027 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:11.191998 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2b197fc-6182-4f88-8e11-d631a9d95b37-istiod-ca-cert\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp\" (UID: \"e2b197fc-6182-4f88-8e11-d631a9d95b37\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp" Nov 01 20:09:11.192027 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:11.192019 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2b197fc-6182-4f88-8e11-d631a9d95b37-workload-certs\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp\" (UID: \"e2b197fc-6182-4f88-8e11-d631a9d95b37\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp" Nov 01 20:09:11.293618 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:11.293582 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2b197fc-6182-4f88-8e11-d631a9d95b37-istio-podinfo\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp\" (UID: \"e2b197fc-6182-4f88-8e11-d631a9d95b37\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp" Nov 01 20:09:11.293618 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:11.293617 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2b197fc-6182-4f88-8e11-d631a9d95b37-istiod-ca-cert\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp\" (UID: \"e2b197fc-6182-4f88-8e11-d631a9d95b37\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp" Nov 01 20:09:11.293864 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:11.293645 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2b197fc-6182-4f88-8e11-d631a9d95b37-workload-certs\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp\" (UID: \"e2b197fc-6182-4f88-8e11-d631a9d95b37\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp" Nov 01 20:09:11.293864 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:11.293671 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2b197fc-6182-4f88-8e11-d631a9d95b37-istio-envoy\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp\" (UID: \"e2b197fc-6182-4f88-8e11-d631a9d95b37\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp" Nov 01 20:09:11.293864 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:11.293768 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2b197fc-6182-4f88-8e11-d631a9d95b37-workload-socket\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp\" (UID: \"e2b197fc-6182-4f88-8e11-d631a9d95b37\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp" Nov 01 20:09:11.293864 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:11.293784 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2b197fc-6182-4f88-8e11-d631a9d95b37-istio-data\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp\" (UID: \"e2b197fc-6182-4f88-8e11-d631a9d95b37\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp" Nov 01 20:09:11.293864 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:11.293815 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2b197fc-6182-4f88-8e11-d631a9d95b37-istio-token\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp\" (UID: \"e2b197fc-6182-4f88-8e11-d631a9d95b37\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp" Nov 01 20:09:11.293864 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:11.293849 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2b197fc-6182-4f88-8e11-d631a9d95b37-credential-socket\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp\" (UID: \"e2b197fc-6182-4f88-8e11-d631a9d95b37\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp" Nov 01 20:09:11.294158 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:11.293901 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhnzz\" (UniqueName: \"kubernetes.io/projected/e2b197fc-6182-4f88-8e11-d631a9d95b37-kube-api-access-nhnzz\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp\" (UID: \"e2b197fc-6182-4f88-8e11-d631a9d95b37\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp" Nov 01 20:09:11.294221 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:11.294203 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2b197fc-6182-4f88-8e11-d631a9d95b37-workload-socket\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp\" (UID: \"e2b197fc-6182-4f88-8e11-d631a9d95b37\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp" Nov 01 20:09:11.294336 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:11.294203 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2b197fc-6182-4f88-8e11-d631a9d95b37-workload-certs\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp\" (UID: \"e2b197fc-6182-4f88-8e11-d631a9d95b37\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp" Nov 01 20:09:11.294470 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:11.294447 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2b197fc-6182-4f88-8e11-d631a9d95b37-istiod-ca-cert\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp\" (UID: \"e2b197fc-6182-4f88-8e11-d631a9d95b37\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp" Nov 01 20:09:11.294526 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:11.294470 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2b197fc-6182-4f88-8e11-d631a9d95b37-istio-data\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp\" (UID: \"e2b197fc-6182-4f88-8e11-d631a9d95b37\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp" Nov 01 20:09:11.294526 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:11.294497 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2b197fc-6182-4f88-8e11-d631a9d95b37-credential-socket\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp\" (UID: \"e2b197fc-6182-4f88-8e11-d631a9d95b37\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp" Nov 01 20:09:11.296281 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:11.296245 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2b197fc-6182-4f88-8e11-d631a9d95b37-istio-envoy\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp\" (UID: \"e2b197fc-6182-4f88-8e11-d631a9d95b37\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp" Nov 01 20:09:11.296632 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:11.296609 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2b197fc-6182-4f88-8e11-d631a9d95b37-istio-podinfo\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp\" (UID: \"e2b197fc-6182-4f88-8e11-d631a9d95b37\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp" Nov 01 20:09:11.301788 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:11.301761 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2b197fc-6182-4f88-8e11-d631a9d95b37-istio-token\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp\" (UID: \"e2b197fc-6182-4f88-8e11-d631a9d95b37\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp" Nov 01 20:09:11.302103 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:11.302084 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhnzz\" (UniqueName: \"kubernetes.io/projected/e2b197fc-6182-4f88-8e11-d631a9d95b37-kube-api-access-nhnzz\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp\" (UID: \"e2b197fc-6182-4f88-8e11-d631a9d95b37\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp" Nov 01 20:09:11.430133 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:11.430055 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp" Nov 01 20:09:11.710174 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:11.710145 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp"] Nov 01 20:09:11.711833 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:09:11.711808 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2b197fc_6182_4f88_8e11_d631a9d95b37.slice/crio-a40d9dc91db3411b28cdd7756486759432539b3940783ef4112976652335015a WatchSource:0}: Error finding container a40d9dc91db3411b28cdd7756486759432539b3940783ef4112976652335015a: Status 404 returned error can't find the container with id a40d9dc91db3411b28cdd7756486759432539b3940783ef4112976652335015a Nov 01 20:09:11.714235 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:11.714201 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:09:11.714378 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:11.714280 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:09:11.714378 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:11.714367 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:09:12.686301 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:12.686247 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp" event={"ID":"e2b197fc-6182-4f88-8e11-d631a9d95b37","Type":"ContainerStarted","Data":"39744e4d2aab8710a4a0b9466ed5a0b95f2168fbc1549dad738a2da1348fde5b"} Nov 01 20:09:12.686684 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:12.686327 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp" event={"ID":"e2b197fc-6182-4f88-8e11-d631a9d95b37","Type":"ContainerStarted","Data":"a40d9dc91db3411b28cdd7756486759432539b3940783ef4112976652335015a"} Nov 01 20:09:12.719138 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:12.718162 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp" podStartSLOduration=1.718142907 podStartE2EDuration="1.718142907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:09:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:09:12.711634288 +0000 UTC m=+598.484794838" watchObservedRunningTime="2026-11-01 20:09:12.718142907 +0000 UTC m=+598.491303425" Nov 01 20:09:13.436319 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:13.435645 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp" Nov 01 20:09:13.456307 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:13.449007 5935 patch_prober.go:29] interesting pod/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 01 20:09:13.456307 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:13.449066 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp" podUID="e2b197fc-6182-4f88-8e11-d631a9d95b37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 01 20:09:14.430422 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:14.430391 5935 patch_prober.go:29] interesting pod/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 01 20:09:14.430793 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:14.430442 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp" podUID="e2b197fc-6182-4f88-8e11-d631a9d95b37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 01 20:09:15.431238 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:15.431204 5935 patch_prober.go:29] interesting pod/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 01 20:09:15.431666 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:15.431254 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp" podUID="e2b197fc-6182-4f88-8e11-d631a9d95b37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 01 20:09:16.430909 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:16.430876 5935 patch_prober.go:29] interesting pod/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 01 20:09:16.431118 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:16.430932 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp" podUID="e2b197fc-6182-4f88-8e11-d631a9d95b37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 01 20:09:17.431045 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:17.431002 5935 patch_prober.go:29] interesting pod/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Nov 01 20:09:17.431577 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:17.431071 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp" podUID="e2b197fc-6182-4f88-8e11-d631a9d95b37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Nov 01 20:09:19.112133 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:19.112097 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp" Nov 01 20:09:19.115060 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:19.112584 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp" Nov 01 20:09:19.115060 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:19.113368 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp" Nov 01 20:09:20.761116 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:20.761085 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-5370bb-648c6878bc-j99qx"] Nov 01 20:09:20.764853 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:20.764836 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-5370bb-648c6878bc-j99qx" Nov 01 20:09:20.770869 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:20.770830 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-5370bb\"" Nov 01 20:09:20.771755 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:20.771720 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-5370bb-dockercfg-pxbmt\"" Nov 01 20:09:20.774186 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:20.774166 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-5370bb-648c6878bc-j99qx"] Nov 01 20:09:20.793730 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:20.793704 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-5370bb-6c5d87878c-kng8d"] Nov 01 20:09:20.798104 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:20.798088 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5370bb-6c5d87878c-kng8d" Nov 01 20:09:20.804723 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:20.804655 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-5370bb-6c5d87878c-kng8d"] Nov 01 20:09:20.850058 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:20.850031 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kpxx\" (UniqueName: \"kubernetes.io/projected/4b4d32cc-fa47-4b9d-9e8b-c39dfd020196-kube-api-access-2kpxx\") pod \"payload-pre-processing-e2e-extmod-b-5370bb-6c5d87878c-kng8d\" (UID: \"4b4d32cc-fa47-4b9d-9e8b-c39dfd020196\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5370bb-6c5d87878c-kng8d" Nov 01 20:09:20.850214 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:20.850151 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04b7431a-dbc2-47b6-92b3-da82b44a1b8f-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-5370bb-648c6878bc-j99qx\" (UID: \"04b7431a-dbc2-47b6-92b3-da82b44a1b8f\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-5370bb-648c6878bc-j99qx" Nov 01 20:09:20.850214 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:20.850177 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b4d32cc-fa47-4b9d-9e8b-c39dfd020196-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-5370bb-6c5d87878c-kng8d\" (UID: \"4b4d32cc-fa47-4b9d-9e8b-c39dfd020196\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5370bb-6c5d87878c-kng8d" Nov 01 20:09:20.850592 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:20.850566 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptccw\" (UniqueName: \"kubernetes.io/projected/04b7431a-dbc2-47b6-92b3-da82b44a1b8f-kube-api-access-ptccw\") pod \"payload-processing-e2e-extmod-b-5370bb-648c6878bc-j99qx\" (UID: \"04b7431a-dbc2-47b6-92b3-da82b44a1b8f\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-5370bb-648c6878bc-j99qx" Nov 01 20:09:20.952270 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:20.952238 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kpxx\" (UniqueName: \"kubernetes.io/projected/4b4d32cc-fa47-4b9d-9e8b-c39dfd020196-kube-api-access-2kpxx\") pod \"payload-pre-processing-e2e-extmod-b-5370bb-6c5d87878c-kng8d\" (UID: \"4b4d32cc-fa47-4b9d-9e8b-c39dfd020196\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5370bb-6c5d87878c-kng8d" Nov 01 20:09:20.952434 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:20.952337 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04b7431a-dbc2-47b6-92b3-da82b44a1b8f-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-5370bb-648c6878bc-j99qx\" (UID: \"04b7431a-dbc2-47b6-92b3-da82b44a1b8f\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-5370bb-648c6878bc-j99qx" Nov 01 20:09:20.952434 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:20.952359 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b4d32cc-fa47-4b9d-9e8b-c39dfd020196-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-5370bb-6c5d87878c-kng8d\" (UID: \"4b4d32cc-fa47-4b9d-9e8b-c39dfd020196\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5370bb-6c5d87878c-kng8d" Nov 01 20:09:20.952434 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:20.952387 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptccw\" (UniqueName: \"kubernetes.io/projected/04b7431a-dbc2-47b6-92b3-da82b44a1b8f-kube-api-access-ptccw\") pod \"payload-processing-e2e-extmod-b-5370bb-648c6878bc-j99qx\" (UID: \"04b7431a-dbc2-47b6-92b3-da82b44a1b8f\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-5370bb-648c6878bc-j99qx" Nov 01 20:09:20.953103 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:20.953083 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04b7431a-dbc2-47b6-92b3-da82b44a1b8f-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-5370bb-648c6878bc-j99qx\" (UID: \"04b7431a-dbc2-47b6-92b3-da82b44a1b8f\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-5370bb-648c6878bc-j99qx" Nov 01 20:09:20.953140 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:20.953093 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b4d32cc-fa47-4b9d-9e8b-c39dfd020196-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-5370bb-6c5d87878c-kng8d\" (UID: \"4b4d32cc-fa47-4b9d-9e8b-c39dfd020196\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5370bb-6c5d87878c-kng8d" Nov 01 20:09:20.973575 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:20.973538 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptccw\" (UniqueName: \"kubernetes.io/projected/04b7431a-dbc2-47b6-92b3-da82b44a1b8f-kube-api-access-ptccw\") pod \"payload-processing-e2e-extmod-b-5370bb-648c6878bc-j99qx\" (UID: \"04b7431a-dbc2-47b6-92b3-da82b44a1b8f\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-5370bb-648c6878bc-j99qx" Nov 01 20:09:20.973708 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:20.973668 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kpxx\" (UniqueName: \"kubernetes.io/projected/4b4d32cc-fa47-4b9d-9e8b-c39dfd020196-kube-api-access-2kpxx\") pod \"payload-pre-processing-e2e-extmod-b-5370bb-6c5d87878c-kng8d\" (UID: \"4b4d32cc-fa47-4b9d-9e8b-c39dfd020196\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5370bb-6c5d87878c-kng8d" Nov 01 20:09:21.081539 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:21.081458 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-5370bb-648c6878bc-j99qx" Nov 01 20:09:21.115522 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:21.115493 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5370bb-6c5d87878c-kng8d" Nov 01 20:09:21.359343 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:21.359320 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-5370bb-648c6878bc-j99qx"] Nov 01 20:09:21.361128 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:09:21.361105 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04b7431a_dbc2_47b6_92b3_da82b44a1b8f.slice/crio-dfd7ae0ff2ac07c3ca700b5f0c254f357aba60dbde22f7091b33d149c972dd74 WatchSource:0}: Error finding container dfd7ae0ff2ac07c3ca700b5f0c254f357aba60dbde22f7091b33d149c972dd74: Status 404 returned error can't find the container with id dfd7ae0ff2ac07c3ca700b5f0c254f357aba60dbde22f7091b33d149c972dd74 Nov 01 20:09:21.405500 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:21.405476 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-5370bb-6c5d87878c-kng8d"] Nov 01 20:09:21.410408 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:09:21.410386 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b4d32cc_fa47_4b9d_9e8b_c39dfd020196.slice/crio-04b0c5b716534151ad4a57f092fad568e32d78a882d98490844f128fe0933868 WatchSource:0}: Error finding container 04b0c5b716534151ad4a57f092fad568e32d78a882d98490844f128fe0933868: Status 404 returned error can't find the container with id 04b0c5b716534151ad4a57f092fad568e32d78a882d98490844f128fe0933868 Nov 01 20:09:21.761493 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:21.761452 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5370bb-6c5d87878c-kng8d" event={"ID":"4b4d32cc-fa47-4b9d-9e8b-c39dfd020196","Type":"ContainerStarted","Data":"0d09088a310336b869185a4926d8b57aad71b987620f5e0eddf5d416c9ac2c90"} Nov 01 20:09:21.761966 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:21.761562 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5370bb-6c5d87878c-kng8d" Nov 01 20:09:21.761966 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:21.761591 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5370bb-6c5d87878c-kng8d" event={"ID":"4b4d32cc-fa47-4b9d-9e8b-c39dfd020196","Type":"ContainerStarted","Data":"04b0c5b716534151ad4a57f092fad568e32d78a882d98490844f128fe0933868"} Nov 01 20:09:21.763017 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:21.762989 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-5370bb-648c6878bc-j99qx" event={"ID":"04b7431a-dbc2-47b6-92b3-da82b44a1b8f","Type":"ContainerStarted","Data":"da1b3779c285509a75104127f4259c714290350f3c184984032c1f4b3285cc74"} Nov 01 20:09:21.763122 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:21.763040 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-5370bb-648c6878bc-j99qx" event={"ID":"04b7431a-dbc2-47b6-92b3-da82b44a1b8f","Type":"ContainerStarted","Data":"dfd7ae0ff2ac07c3ca700b5f0c254f357aba60dbde22f7091b33d149c972dd74"} Nov 01 20:09:21.763159 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:21.763142 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-5370bb-648c6878bc-j99qx" Nov 01 20:09:21.778646 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:21.778606 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5370bb-6c5d87878c-kng8d" podStartSLOduration=1.778593632 podStartE2EDuration="1.778593632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:09:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:09:21.776193017 +0000 UTC m=+607.549353535" watchObservedRunningTime="2026-11-01 20:09:21.778593632 +0000 UTC m=+607.551754189" Nov 01 20:09:21.794794 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:21.794749 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-5370bb-648c6878bc-j99qx" podStartSLOduration=1.794735476 podStartE2EDuration="1.794735476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:09:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:09:21.791660944 +0000 UTC m=+607.564821467" watchObservedRunningTime="2026-11-01 20:09:21.794735476 +0000 UTC m=+607.567895988" Nov 01 20:09:25.563306 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:25.562320 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-5370bb-6f959ddc8c-l67b6"] Nov 01 20:09:25.569409 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:25.567539 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5370bb-6f959ddc8c-l67b6" Nov 01 20:09:25.587323 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:25.585981 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-5370bb-6f959ddc8c-l67b6"] Nov 01 20:09:25.587323 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:25.586071 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-5370bb-serving-cert\"" Nov 01 20:09:25.736317 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:25.733945 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32159d48-5e0e-4841-a448-c57d21ab523f-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-5370bb-6f959ddc8c-l67b6\" (UID: \"32159d48-5e0e-4841-a448-c57d21ab523f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5370bb-6f959ddc8c-l67b6" Nov 01 20:09:25.736317 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:25.734065 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32159d48-5e0e-4841-a448-c57d21ab523f-maas-api-tls\") pod \"maas-api-e2e-extmod-b-5370bb-6f959ddc8c-l67b6\" (UID: \"32159d48-5e0e-4841-a448-c57d21ab523f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5370bb-6f959ddc8c-l67b6" Nov 01 20:09:25.736317 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:25.734153 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2wxd\" (UniqueName: \"kubernetes.io/projected/32159d48-5e0e-4841-a448-c57d21ab523f-kube-api-access-s2wxd\") pod \"maas-api-e2e-extmod-b-5370bb-6f959ddc8c-l67b6\" (UID: \"32159d48-5e0e-4841-a448-c57d21ab523f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5370bb-6f959ddc8c-l67b6" Nov 01 20:09:25.736317 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:25.734187 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32159d48-5e0e-4841-a448-c57d21ab523f-metrics-certs\") pod \"maas-api-e2e-extmod-b-5370bb-6f959ddc8c-l67b6\" (UID: \"32159d48-5e0e-4841-a448-c57d21ab523f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5370bb-6f959ddc8c-l67b6" Nov 01 20:09:25.837371 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:25.836921 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32159d48-5e0e-4841-a448-c57d21ab523f-maas-api-tls\") pod \"maas-api-e2e-extmod-b-5370bb-6f959ddc8c-l67b6\" (UID: \"32159d48-5e0e-4841-a448-c57d21ab523f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5370bb-6f959ddc8c-l67b6" Nov 01 20:09:25.837371 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:25.837028 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2wxd\" (UniqueName: \"kubernetes.io/projected/32159d48-5e0e-4841-a448-c57d21ab523f-kube-api-access-s2wxd\") pod \"maas-api-e2e-extmod-b-5370bb-6f959ddc8c-l67b6\" (UID: \"32159d48-5e0e-4841-a448-c57d21ab523f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5370bb-6f959ddc8c-l67b6" Nov 01 20:09:25.837371 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:25.837065 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32159d48-5e0e-4841-a448-c57d21ab523f-metrics-certs\") pod \"maas-api-e2e-extmod-b-5370bb-6f959ddc8c-l67b6\" (UID: \"32159d48-5e0e-4841-a448-c57d21ab523f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5370bb-6f959ddc8c-l67b6" Nov 01 20:09:25.837371 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:25.837171 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32159d48-5e0e-4841-a448-c57d21ab523f-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-5370bb-6f959ddc8c-l67b6\" (UID: \"32159d48-5e0e-4841-a448-c57d21ab523f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5370bb-6f959ddc8c-l67b6" Nov 01 20:09:25.838320 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:25.838048 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32159d48-5e0e-4841-a448-c57d21ab523f-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-5370bb-6f959ddc8c-l67b6\" (UID: \"32159d48-5e0e-4841-a448-c57d21ab523f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5370bb-6f959ddc8c-l67b6" Nov 01 20:09:25.838320 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:09:25.838180 5935 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-5370bb-serving-cert: secret "maas-api-e2e-extmod-b-5370bb-serving-cert" not found Nov 01 20:09:25.838320 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:09:25.838266 5935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32159d48-5e0e-4841-a448-c57d21ab523f-maas-api-tls podName:32159d48-5e0e-4841-a448-c57d21ab523f nodeName:}" failed. No retries permitted until 2026-11-01 20:09:26.338238878 +0000 UTC m=+612.111399386 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/32159d48-5e0e-4841-a448-c57d21ab523f-maas-api-tls") pod "maas-api-e2e-extmod-b-5370bb-6f959ddc8c-l67b6" (UID: "32159d48-5e0e-4841-a448-c57d21ab523f") : secret "maas-api-e2e-extmod-b-5370bb-serving-cert" not found Nov 01 20:09:25.852309 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:25.842082 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32159d48-5e0e-4841-a448-c57d21ab523f-metrics-certs\") pod \"maas-api-e2e-extmod-b-5370bb-6f959ddc8c-l67b6\" (UID: \"32159d48-5e0e-4841-a448-c57d21ab523f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5370bb-6f959ddc8c-l67b6" Nov 01 20:09:25.873306 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:25.868022 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2wxd\" (UniqueName: \"kubernetes.io/projected/32159d48-5e0e-4841-a448-c57d21ab523f-kube-api-access-s2wxd\") pod \"maas-api-e2e-extmod-b-5370bb-6f959ddc8c-l67b6\" (UID: \"32159d48-5e0e-4841-a448-c57d21ab523f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5370bb-6f959ddc8c-l67b6" Nov 01 20:09:26.018311 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.017267 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s"] Nov 01 20:09:26.026186 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.026160 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s" Nov 01 20:09:26.043449 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.042681 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s"] Nov 01 20:09:26.071759 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.068808 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s"] Nov 01 20:09:26.406068 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.403160 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8bf3d0d-c668-44f1-9238-746e548faba0-istio-data\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s\" (UID: \"e8bf3d0d-c668-44f1-9238-746e548faba0\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s" Nov 01 20:09:26.406068 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.403221 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32159d48-5e0e-4841-a448-c57d21ab523f-maas-api-tls\") pod \"maas-api-e2e-extmod-b-5370bb-6f959ddc8c-l67b6\" (UID: \"32159d48-5e0e-4841-a448-c57d21ab523f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5370bb-6f959ddc8c-l67b6" Nov 01 20:09:26.406068 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.403257 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bf3d0d-c668-44f1-9238-746e548faba0-workload-socket\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s\" (UID: \"e8bf3d0d-c668-44f1-9238-746e548faba0\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s" Nov 01 20:09:26.406068 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.403331 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8bf3d0d-c668-44f1-9238-746e548faba0-workload-certs\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s\" (UID: \"e8bf3d0d-c668-44f1-9238-746e548faba0\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s" Nov 01 20:09:26.406068 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.403370 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bf3d0d-c668-44f1-9238-746e548faba0-credential-socket\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s\" (UID: \"e8bf3d0d-c668-44f1-9238-746e548faba0\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s" Nov 01 20:09:26.406068 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.403402 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8bf3d0d-c668-44f1-9238-746e548faba0-istio-envoy\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s\" (UID: \"e8bf3d0d-c668-44f1-9238-746e548faba0\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s" Nov 01 20:09:26.406068 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.403480 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6vzt\" (UniqueName: \"kubernetes.io/projected/e8bf3d0d-c668-44f1-9238-746e548faba0-kube-api-access-h6vzt\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s\" (UID: \"e8bf3d0d-c668-44f1-9238-746e548faba0\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s" Nov 01 20:09:26.406068 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.403512 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8bf3d0d-c668-44f1-9238-746e548faba0-istio-token\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s\" (UID: \"e8bf3d0d-c668-44f1-9238-746e548faba0\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s" Nov 01 20:09:26.406068 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.403539 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8bf3d0d-c668-44f1-9238-746e548faba0-istiod-ca-cert\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s\" (UID: \"e8bf3d0d-c668-44f1-9238-746e548faba0\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s" Nov 01 20:09:26.406068 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.403635 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8bf3d0d-c668-44f1-9238-746e548faba0-istio-podinfo\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s\" (UID: \"e8bf3d0d-c668-44f1-9238-746e548faba0\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s" Nov 01 20:09:26.406068 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:09:26.404759 5935 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-5370bb-serving-cert: secret "maas-api-e2e-extmod-b-5370bb-serving-cert" not found Nov 01 20:09:26.406068 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:09:26.404825 5935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32159d48-5e0e-4841-a448-c57d21ab523f-maas-api-tls podName:32159d48-5e0e-4841-a448-c57d21ab523f nodeName:}" failed. No retries permitted until 2026-11-01 20:09:27.404805246 +0000 UTC m=+613.177965743 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/32159d48-5e0e-4841-a448-c57d21ab523f-maas-api-tls") pod "maas-api-e2e-extmod-b-5370bb-6f959ddc8c-l67b6" (UID: "32159d48-5e0e-4841-a448-c57d21ab523f") : secret "maas-api-e2e-extmod-b-5370bb-serving-cert" not found Nov 01 20:09:26.417645 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.417606 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s" Nov 01 20:09:26.427746 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.427714 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s"] Nov 01 20:09:26.427896 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.427820 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd"] Nov 01 20:09:26.432695 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.432671 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd"] Nov 01 20:09:26.432815 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.432733 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k"] Nov 01 20:09:26.434438 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.434415 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd" Nov 01 20:09:26.445703 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.445680 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k"] Nov 01 20:09:26.445838 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.445783 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k" Nov 01 20:09:26.504673 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.504642 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8686280e-385b-4a98-a1d2-5018e5f08fc1-credential-socket\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd\" (UID: \"8686280e-385b-4a98-a1d2-5018e5f08fc1\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd" Nov 01 20:09:26.504845 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.504677 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8bf3d0d-c668-44f1-9238-746e548faba0-istio-podinfo\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s\" (UID: \"e8bf3d0d-c668-44f1-9238-746e548faba0\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s" Nov 01 20:09:26.504845 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.504754 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a-istio-podinfo\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k\" (UID: \"7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k" Nov 01 20:09:26.504987 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.504866 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a-istio-token\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k\" (UID: \"7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k" Nov 01 20:09:26.504987 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.504882 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8686280e-385b-4a98-a1d2-5018e5f08fc1-istio-envoy\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd\" (UID: \"8686280e-385b-4a98-a1d2-5018e5f08fc1\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd" Nov 01 20:09:26.504987 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.504902 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8bf3d0d-c668-44f1-9238-746e548faba0-istio-data\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s\" (UID: \"e8bf3d0d-c668-44f1-9238-746e548faba0\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s" Nov 01 20:09:26.504987 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.504918 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8686280e-385b-4a98-a1d2-5018e5f08fc1-istio-token\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd\" (UID: \"8686280e-385b-4a98-a1d2-5018e5f08fc1\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd" Nov 01 20:09:26.505183 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.505039 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bf3d0d-c668-44f1-9238-746e548faba0-workload-socket\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s\" (UID: \"e8bf3d0d-c668-44f1-9238-746e548faba0\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s" Nov 01 20:09:26.505183 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.505082 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8686280e-385b-4a98-a1d2-5018e5f08fc1-istio-podinfo\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd\" (UID: \"8686280e-385b-4a98-a1d2-5018e5f08fc1\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd" Nov 01 20:09:26.505298 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.505257 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8bf3d0d-c668-44f1-9238-746e548faba0-istio-data\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s\" (UID: \"e8bf3d0d-c668-44f1-9238-746e548faba0\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s" Nov 01 20:09:26.505298 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.505268 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae-workload-socket\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s\" (UID: \"b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s" Nov 01 20:09:26.505392 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.505351 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae-istio-envoy\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s\" (UID: \"b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s" Nov 01 20:09:26.505392 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.505375 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bf3d0d-c668-44f1-9238-746e548faba0-workload-socket\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s\" (UID: \"e8bf3d0d-c668-44f1-9238-746e548faba0\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s" Nov 01 20:09:26.505392 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.505385 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae-istio-data\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s\" (UID: \"b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s" Nov 01 20:09:26.505518 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.505490 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae-istiod-ca-cert\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s\" (UID: \"b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s" Nov 01 20:09:26.505556 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.505516 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8bf3d0d-c668-44f1-9238-746e548faba0-workload-certs\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s\" (UID: \"e8bf3d0d-c668-44f1-9238-746e548faba0\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s" Nov 01 20:09:26.505556 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.505542 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8686280e-385b-4a98-a1d2-5018e5f08fc1-istio-data\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd\" (UID: \"8686280e-385b-4a98-a1d2-5018e5f08fc1\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd" Nov 01 20:09:26.505658 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.505595 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bf3d0d-c668-44f1-9238-746e548faba0-credential-socket\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s\" (UID: \"e8bf3d0d-c668-44f1-9238-746e548faba0\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s" Nov 01 20:09:26.505658 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.505618 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dn86x\" (UniqueName: \"kubernetes.io/projected/8686280e-385b-4a98-a1d2-5018e5f08fc1-kube-api-access-dn86x\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd\" (UID: \"8686280e-385b-4a98-a1d2-5018e5f08fc1\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd" Nov 01 20:09:26.505658 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.505641 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8bf3d0d-c668-44f1-9238-746e548faba0-istio-envoy\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s\" (UID: \"e8bf3d0d-c668-44f1-9238-746e548faba0\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s" Nov 01 20:09:26.505824 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.505664 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a-workload-certs\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k\" (UID: \"7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k" Nov 01 20:09:26.505824 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.505707 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae-workload-certs\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s\" (UID: \"b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s" Nov 01 20:09:26.505824 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.505789 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae-istio-podinfo\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s\" (UID: \"b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s" Nov 01 20:09:26.505986 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.505858 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8bf3d0d-c668-44f1-9238-746e548faba0-workload-certs\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s\" (UID: \"e8bf3d0d-c668-44f1-9238-746e548faba0\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s" Nov 01 20:09:26.505986 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.505870 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae-credential-socket\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s\" (UID: \"b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s" Nov 01 20:09:26.505986 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.505928 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8686280e-385b-4a98-a1d2-5018e5f08fc1-istiod-ca-cert\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd\" (UID: \"8686280e-385b-4a98-a1d2-5018e5f08fc1\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd" Nov 01 20:09:26.505986 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.505932 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bf3d0d-c668-44f1-9238-746e548faba0-credential-socket\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s\" (UID: \"e8bf3d0d-c668-44f1-9238-746e548faba0\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s" Nov 01 20:09:26.506165 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.506134 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a-istiod-ca-cert\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k\" (UID: \"7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k" Nov 01 20:09:26.506212 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.506173 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a-workload-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k\" (UID: \"7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k" Nov 01 20:09:26.506264 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.506220 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a-istio-data\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k\" (UID: \"7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k" Nov 01 20:09:26.506359 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.506342 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6vzt\" (UniqueName: \"kubernetes.io/projected/e8bf3d0d-c668-44f1-9238-746e548faba0-kube-api-access-h6vzt\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s\" (UID: \"e8bf3d0d-c668-44f1-9238-746e548faba0\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s" Nov 01 20:09:26.506448 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.506427 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8686280e-385b-4a98-a1d2-5018e5f08fc1-workload-certs\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd\" (UID: \"8686280e-385b-4a98-a1d2-5018e5f08fc1\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd" Nov 01 20:09:26.506526 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.506496 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a-credential-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k\" (UID: \"7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k" Nov 01 20:09:26.506593 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.506578 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8bf3d0d-c668-44f1-9238-746e548faba0-istio-token\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s\" (UID: \"e8bf3d0d-c668-44f1-9238-746e548faba0\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s" Nov 01 20:09:26.506647 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.506606 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8bf3d0d-c668-44f1-9238-746e548faba0-istiod-ca-cert\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s\" (UID: \"e8bf3d0d-c668-44f1-9238-746e548faba0\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s" Nov 01 20:09:26.506647 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.506633 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a-istio-envoy\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k\" (UID: \"7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k" Nov 01 20:09:26.506755 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.506648 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae-istio-token\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s\" (UID: \"b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s" Nov 01 20:09:26.507168 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.507145 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8bf3d0d-c668-44f1-9238-746e548faba0-istiod-ca-cert\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s\" (UID: \"e8bf3d0d-c668-44f1-9238-746e548faba0\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s" Nov 01 20:09:26.507321 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.507183 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s6jg\" (UniqueName: \"kubernetes.io/projected/b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae-kube-api-access-2s6jg\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s\" (UID: \"b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s" Nov 01 20:09:26.507321 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.507206 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg4ks\" (UniqueName: \"kubernetes.io/projected/7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a-kube-api-access-lg4ks\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k\" (UID: \"7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k" Nov 01 20:09:26.507321 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.507240 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8686280e-385b-4a98-a1d2-5018e5f08fc1-workload-socket\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd\" (UID: \"8686280e-385b-4a98-a1d2-5018e5f08fc1\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd" Nov 01 20:09:26.507723 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.507698 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8bf3d0d-c668-44f1-9238-746e548faba0-istio-podinfo\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s\" (UID: \"e8bf3d0d-c668-44f1-9238-746e548faba0\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s" Nov 01 20:09:26.508055 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.508038 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8bf3d0d-c668-44f1-9238-746e548faba0-istio-envoy\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s\" (UID: \"e8bf3d0d-c668-44f1-9238-746e548faba0\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s" Nov 01 20:09:26.516046 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.516023 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8bf3d0d-c668-44f1-9238-746e548faba0-istio-token\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s\" (UID: \"e8bf3d0d-c668-44f1-9238-746e548faba0\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s" Nov 01 20:09:26.516202 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.516184 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6vzt\" (UniqueName: \"kubernetes.io/projected/e8bf3d0d-c668-44f1-9238-746e548faba0-kube-api-access-h6vzt\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s\" (UID: \"e8bf3d0d-c668-44f1-9238-746e548faba0\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s" Nov 01 20:09:26.607888 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.607828 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8686280e-385b-4a98-a1d2-5018e5f08fc1-istio-token\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd\" (UID: \"8686280e-385b-4a98-a1d2-5018e5f08fc1\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd" Nov 01 20:09:26.607888 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.607875 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8686280e-385b-4a98-a1d2-5018e5f08fc1-istio-podinfo\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd\" (UID: \"8686280e-385b-4a98-a1d2-5018e5f08fc1\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd" Nov 01 20:09:26.608350 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.607897 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae-workload-socket\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s\" (UID: \"b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s" Nov 01 20:09:26.608350 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.607914 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae-istio-envoy\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s\" (UID: \"b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s" Nov 01 20:09:26.608350 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.607927 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae-istio-data\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s\" (UID: \"b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s" Nov 01 20:09:26.608350 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.607958 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae-istiod-ca-cert\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s\" (UID: \"b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s" Nov 01 20:09:26.608350 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.607987 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8686280e-385b-4a98-a1d2-5018e5f08fc1-istio-data\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd\" (UID: \"8686280e-385b-4a98-a1d2-5018e5f08fc1\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd" Nov 01 20:09:26.608350 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.608046 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dn86x\" (UniqueName: \"kubernetes.io/projected/8686280e-385b-4a98-a1d2-5018e5f08fc1-kube-api-access-dn86x\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd\" (UID: \"8686280e-385b-4a98-a1d2-5018e5f08fc1\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd" Nov 01 20:09:26.608350 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.608098 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a-workload-certs\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k\" (UID: \"7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k" Nov 01 20:09:26.608350 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.608126 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae-workload-certs\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s\" (UID: \"b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s" Nov 01 20:09:26.608350 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.608159 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae-istio-podinfo\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s\" (UID: \"b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s" Nov 01 20:09:26.608350 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.608186 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae-credential-socket\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s\" (UID: \"b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s" Nov 01 20:09:26.608350 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.608218 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8686280e-385b-4a98-a1d2-5018e5f08fc1-istiod-ca-cert\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd\" (UID: \"8686280e-385b-4a98-a1d2-5018e5f08fc1\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd" Nov 01 20:09:26.608350 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.608270 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a-istiod-ca-cert\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k\" (UID: \"7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k" Nov 01 20:09:26.608350 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.608311 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a-workload-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k\" (UID: \"7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k" Nov 01 20:09:26.608350 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.608347 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae-workload-socket\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s\" (UID: \"b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s" Nov 01 20:09:26.608350 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.608360 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a-istio-data\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k\" (UID: \"7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k" Nov 01 20:09:26.609140 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.608375 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae-istio-data\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s\" (UID: \"b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s" Nov 01 20:09:26.609140 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.608403 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8686280e-385b-4a98-a1d2-5018e5f08fc1-workload-certs\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd\" (UID: \"8686280e-385b-4a98-a1d2-5018e5f08fc1\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd" Nov 01 20:09:26.609140 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.608438 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a-credential-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k\" (UID: \"7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k" Nov 01 20:09:26.609140 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.608494 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a-istio-envoy\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k\" (UID: \"7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k" Nov 01 20:09:26.609140 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.608521 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae-istio-token\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s\" (UID: \"b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s" Nov 01 20:09:26.609140 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.608552 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s6jg\" (UniqueName: \"kubernetes.io/projected/b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae-kube-api-access-2s6jg\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s\" (UID: \"b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s" Nov 01 20:09:26.609140 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.608585 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg4ks\" (UniqueName: \"kubernetes.io/projected/7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a-kube-api-access-lg4ks\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k\" (UID: \"7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k" Nov 01 20:09:26.609140 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.608625 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8686280e-385b-4a98-a1d2-5018e5f08fc1-workload-socket\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd\" (UID: \"8686280e-385b-4a98-a1d2-5018e5f08fc1\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd" Nov 01 20:09:26.609140 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.608683 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8686280e-385b-4a98-a1d2-5018e5f08fc1-credential-socket\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd\" (UID: \"8686280e-385b-4a98-a1d2-5018e5f08fc1\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd" Nov 01 20:09:26.609140 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.608728 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a-istio-data\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k\" (UID: \"7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k" Nov 01 20:09:26.609140 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.608735 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8686280e-385b-4a98-a1d2-5018e5f08fc1-istio-data\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd\" (UID: \"8686280e-385b-4a98-a1d2-5018e5f08fc1\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd" Nov 01 20:09:26.609140 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.608756 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a-istio-podinfo\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k\" (UID: \"7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k" Nov 01 20:09:26.609140 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.608994 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a-workload-certs\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k\" (UID: \"7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k" Nov 01 20:09:26.609140 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.609102 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae-istiod-ca-cert\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s\" (UID: \"b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s" Nov 01 20:09:26.609140 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.609114 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8686280e-385b-4a98-a1d2-5018e5f08fc1-workload-certs\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd\" (UID: \"8686280e-385b-4a98-a1d2-5018e5f08fc1\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd" Nov 01 20:09:26.609924 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.609195 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae-workload-certs\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s\" (UID: \"b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s" Nov 01 20:09:26.609924 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.609257 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a-credential-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k\" (UID: \"7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k" Nov 01 20:09:26.609924 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.609618 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8686280e-385b-4a98-a1d2-5018e5f08fc1-workload-socket\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd\" (UID: \"8686280e-385b-4a98-a1d2-5018e5f08fc1\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd" Nov 01 20:09:26.609924 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.609702 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a-workload-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k\" (UID: \"7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k" Nov 01 20:09:26.611038 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.610364 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8686280e-385b-4a98-a1d2-5018e5f08fc1-istiod-ca-cert\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd\" (UID: \"8686280e-385b-4a98-a1d2-5018e5f08fc1\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd" Nov 01 20:09:26.611038 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.610455 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8686280e-385b-4a98-a1d2-5018e5f08fc1-credential-socket\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd\" (UID: \"8686280e-385b-4a98-a1d2-5018e5f08fc1\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd" Nov 01 20:09:26.611038 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.610672 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae-credential-socket\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s\" (UID: \"b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s" Nov 01 20:09:26.611038 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.610714 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae-istio-envoy\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s\" (UID: \"b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s" Nov 01 20:09:26.611038 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.610763 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a-istio-token\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k\" (UID: \"7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k" Nov 01 20:09:26.611038 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.610795 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8686280e-385b-4a98-a1d2-5018e5f08fc1-istio-envoy\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd\" (UID: \"8686280e-385b-4a98-a1d2-5018e5f08fc1\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd" Nov 01 20:09:26.611038 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.611011 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8686280e-385b-4a98-a1d2-5018e5f08fc1-istio-podinfo\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd\" (UID: \"8686280e-385b-4a98-a1d2-5018e5f08fc1\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd" Nov 01 20:09:26.611530 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.611373 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a-istiod-ca-cert\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k\" (UID: \"7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k" Nov 01 20:09:26.611616 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.611593 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a-istio-envoy\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k\" (UID: \"7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k" Nov 01 20:09:26.611947 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.611898 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae-istio-podinfo\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s\" (UID: \"b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s" Nov 01 20:09:26.612996 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.612974 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a-istio-podinfo\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k\" (UID: \"7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k" Nov 01 20:09:26.613063 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.613020 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8686280e-385b-4a98-a1d2-5018e5f08fc1-istio-envoy\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd\" (UID: \"8686280e-385b-4a98-a1d2-5018e5f08fc1\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd" Nov 01 20:09:26.616386 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.616366 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae-istio-token\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s\" (UID: \"b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s" Nov 01 20:09:26.616475 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.616411 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dn86x\" (UniqueName: \"kubernetes.io/projected/8686280e-385b-4a98-a1d2-5018e5f08fc1-kube-api-access-dn86x\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd\" (UID: \"8686280e-385b-4a98-a1d2-5018e5f08fc1\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd" Nov 01 20:09:26.617108 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.617083 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8686280e-385b-4a98-a1d2-5018e5f08fc1-istio-token\") pod \"e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd\" (UID: \"8686280e-385b-4a98-a1d2-5018e5f08fc1\") " pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd" Nov 01 20:09:26.617415 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.617393 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg4ks\" (UniqueName: \"kubernetes.io/projected/7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a-kube-api-access-lg4ks\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k\" (UID: \"7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k" Nov 01 20:09:26.618060 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.618039 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a-istio-token\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k\" (UID: \"7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k" Nov 01 20:09:26.618543 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.618524 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s6jg\" (UniqueName: \"kubernetes.io/projected/b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae-kube-api-access-2s6jg\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s\" (UID: \"b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s" Nov 01 20:09:26.711781 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.711759 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s" Nov 01 20:09:26.739228 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.739206 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s" Nov 01 20:09:26.751162 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.751136 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd" Nov 01 20:09:26.760209 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.760184 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k" Nov 01 20:09:26.995990 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.995965 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s"] Nov 01 20:09:26.996890 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:09:26.996862 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8bf3d0d_c668_44f1_9238_746e548faba0.slice/crio-074d62fd641901f6faf175995083fbeb89daee707c1dd5568aeb4546feb8a899 WatchSource:0}: Error finding container 074d62fd641901f6faf175995083fbeb89daee707c1dd5568aeb4546feb8a899: Status 404 returned error can't find the container with id 074d62fd641901f6faf175995083fbeb89daee707c1dd5568aeb4546feb8a899 Nov 01 20:09:26.999255 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.999222 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:09:26.999370 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.999297 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:09:26.999370 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:26.999334 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:09:27.039719 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:27.039692 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s"] Nov 01 20:09:27.040407 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:09:27.040383 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb52dfb6f_d9c7_4b1f_9cca_13d67c4758ae.slice/crio-e9842ed11234cef35e48d2de47a3c66cd884efb203330d0ea0caf1cc08742fb4 WatchSource:0}: Error finding container e9842ed11234cef35e48d2de47a3c66cd884efb203330d0ea0caf1cc08742fb4: Status 404 returned error can't find the container with id e9842ed11234cef35e48d2de47a3c66cd884efb203330d0ea0caf1cc08742fb4 Nov 01 20:09:27.043798 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:27.043737 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:09:27.043798 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:27.043791 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:09:27.043908 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:27.043819 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:09:27.045251 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:27.045226 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd"] Nov 01 20:09:27.046356 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:09:27.046336 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8686280e_385b_4a98_a1d2_5018e5f08fc1.slice/crio-377a07338d3eb72d60b2dd38594e9b157855383457987860454957d1b548050b WatchSource:0}: Error finding container 377a07338d3eb72d60b2dd38594e9b157855383457987860454957d1b548050b: Status 404 returned error can't find the container with id 377a07338d3eb72d60b2dd38594e9b157855383457987860454957d1b548050b Nov 01 20:09:27.048545 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:27.048514 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:09:27.048612 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:27.048579 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:09:27.048681 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:27.048616 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:09:27.066719 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:27.066695 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k"] Nov 01 20:09:27.073427 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:09:27.073405 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f4577ff_f9ab_4b7b_b1a3_3d66f31cdb7a.slice/crio-995c5edb50fd55762876dd7222f422c9ae45cc9ed83091c080d8469cea3eb7f8 WatchSource:0}: Error finding container 995c5edb50fd55762876dd7222f422c9ae45cc9ed83091c080d8469cea3eb7f8: Status 404 returned error can't find the container with id 995c5edb50fd55762876dd7222f422c9ae45cc9ed83091c080d8469cea3eb7f8 Nov 01 20:09:27.076467 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:27.076438 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:09:27.076565 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:27.076511 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:09:27.076565 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:27.076557 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:09:27.428233 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:27.428141 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32159d48-5e0e-4841-a448-c57d21ab523f-maas-api-tls\") pod \"maas-api-e2e-extmod-b-5370bb-6f959ddc8c-l67b6\" (UID: \"32159d48-5e0e-4841-a448-c57d21ab523f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5370bb-6f959ddc8c-l67b6" Nov 01 20:09:27.431298 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:27.431261 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32159d48-5e0e-4841-a448-c57d21ab523f-maas-api-tls\") pod \"maas-api-e2e-extmod-b-5370bb-6f959ddc8c-l67b6\" (UID: \"32159d48-5e0e-4841-a448-c57d21ab523f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5370bb-6f959ddc8c-l67b6" Nov 01 20:09:27.703414 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:27.703387 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5370bb-6f959ddc8c-l67b6" Nov 01 20:09:27.856558 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:27.856518 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd" event={"ID":"8686280e-385b-4a98-a1d2-5018e5f08fc1","Type":"ContainerStarted","Data":"186f5408970ca77b454a3ad7ec7b60100fe1e84fd96e75d6232dad91ab56ba9e"} Nov 01 20:09:27.856767 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:27.856588 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd" event={"ID":"8686280e-385b-4a98-a1d2-5018e5f08fc1","Type":"ContainerStarted","Data":"377a07338d3eb72d60b2dd38594e9b157855383457987860454957d1b548050b"} Nov 01 20:09:27.860337 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:27.859620 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s" event={"ID":"e8bf3d0d-c668-44f1-9238-746e548faba0","Type":"ContainerStarted","Data":"d9f20bfbfd75bb81796734a93a6ea4e1d1b6b0769209f29666991739058cfdc8"} Nov 01 20:09:27.860337 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:27.859673 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s" event={"ID":"e8bf3d0d-c668-44f1-9238-746e548faba0","Type":"ContainerStarted","Data":"074d62fd641901f6faf175995083fbeb89daee707c1dd5568aeb4546feb8a899"} Nov 01 20:09:27.863124 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:27.862535 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s" event={"ID":"b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae","Type":"ContainerStarted","Data":"9d37d4afbd227de761ef4dde23cedcf9d9a46527ba784e027d6d8d49548bf6c4"} Nov 01 20:09:27.863124 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:27.862578 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s" event={"ID":"b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae","Type":"ContainerStarted","Data":"e9842ed11234cef35e48d2de47a3c66cd884efb203330d0ea0caf1cc08742fb4"} Nov 01 20:09:27.869954 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:27.869918 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k" event={"ID":"7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a","Type":"ContainerStarted","Data":"c7b40909a9d6f1c85c274b821cd3a83987377e042d04e96d01bd1e7f89ba2305"} Nov 01 20:09:27.870177 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:27.870155 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k" event={"ID":"7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a","Type":"ContainerStarted","Data":"995c5edb50fd55762876dd7222f422c9ae45cc9ed83091c080d8469cea3eb7f8"} Nov 01 20:09:28.026383 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:28.024790 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd" podStartSLOduration=2.024771012 podStartE2EDuration="2.024771012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:09:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:09:27.999454646 +0000 UTC m=+613.772615159" watchObservedRunningTime="2026-11-01 20:09:28.024771012 +0000 UTC m=+613.797931524" Nov 01 20:09:28.182564 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:28.177250 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s" podStartSLOduration=2.177229324 podStartE2EDuration="2.177229324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:09:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:09:28.061888953 +0000 UTC m=+613.835049469" watchObservedRunningTime="2026-11-01 20:09:28.177229324 +0000 UTC m=+613.950389836" Nov 01 20:09:28.182869 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:28.182783 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-5370bb-6f959ddc8c-l67b6"] Nov 01 20:09:28.247310 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:28.247177 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k" podStartSLOduration=2.247156666 podStartE2EDuration="2.247156666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:09:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:09:28.131794016 +0000 UTC m=+613.904954529" watchObservedRunningTime="2026-11-01 20:09:28.247156666 +0000 UTC m=+614.020317174" Nov 01 20:09:28.347366 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:28.321602 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s" podStartSLOduration=3.321577503 podStartE2EDuration="3.321577503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:09:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:09:28.198637442 +0000 UTC m=+613.971797956" watchObservedRunningTime="2026-11-01 20:09:28.321577503 +0000 UTC m=+614.094738016" Nov 01 20:09:28.714313 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:28.712878 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s" Nov 01 20:09:28.728086 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:28.727720 5935 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 01 20:09:28.728086 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:28.727784 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s" podUID="e8bf3d0d-c668-44f1-9238-746e548faba0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 01 20:09:28.768557 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:28.766875 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd" Nov 01 20:09:28.768557 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:28.766994 5935 patch_prober.go:29] interesting pod/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 01 20:09:28.768557 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:28.767039 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd" podUID="8686280e-385b-4a98-a1d2-5018e5f08fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 01 20:09:28.769098 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:28.768897 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k" Nov 01 20:09:28.769098 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:28.768948 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s" Nov 01 20:09:28.769098 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:28.769014 5935 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 01 20:09:28.769098 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:28.769045 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k" podUID="7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 01 20:09:28.809461 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:28.778939 5935 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 01 20:09:28.809461 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:28.778988 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s" podUID="b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 01 20:09:29.104593 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:29.104480 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5370bb-6f959ddc8c-l67b6" event={"ID":"32159d48-5e0e-4841-a448-c57d21ab523f","Type":"ContainerStarted","Data":"da86dede87a3d22867878eb5f2baf095304dfc0f35a3e318ae22945d6fb897e5"} Nov 01 20:09:29.720310 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:29.712640 5935 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 01 20:09:29.720310 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:29.712698 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s" podUID="e8bf3d0d-c668-44f1-9238-746e548faba0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 01 20:09:29.754309 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:29.750388 5935 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 01 20:09:29.754309 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:29.750463 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s" podUID="b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 01 20:09:29.770306 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:29.766385 5935 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 01 20:09:29.770306 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:29.766440 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k" podUID="7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 01 20:09:29.770306 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:29.766764 5935 patch_prober.go:29] interesting pod/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 01 20:09:29.770306 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:29.766800 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd" podUID="8686280e-385b-4a98-a1d2-5018e5f08fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 01 20:09:29.953309 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:29.952259 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/simulator-w4-0aa61b-kserve-6774594d4b-qmjz8" podUID="09414dc0-7742-48fa-a4b5-571199d99e15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 20:09:30.141378 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:30.141146 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5370bb-6f959ddc8c-l67b6" event={"ID":"32159d48-5e0e-4841-a448-c57d21ab523f","Type":"ContainerStarted","Data":"aad073906bb2c4b404c8045eb809f4a5484c4f14eddca678c9b20cd5188dc05d"} Nov 01 20:09:30.144323 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:30.142105 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5370bb-6f959ddc8c-l67b6" Nov 01 20:09:30.185324 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:30.182362 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5370bb-6f959ddc8c-l67b6" podStartSLOduration=4.204467492 podStartE2EDuration="5.182340107s" totalImagesPullingTime="977.872615ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:09:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:09:30.181770387 +0000 UTC m=+615.954930924" watchObservedRunningTime="2026-11-01 20:09:30.182340107 +0000 UTC m=+615.955500620" Nov 01 20:09:30.246430 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:30.246376 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-e3c4a7/test-model-e3c4a7-kserve-59c98dffb5-cqbbk" podUID="b5df4b03-d387-44c7-a263-e24f9730ed0e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": context deadline exceeded" Nov 01 20:09:30.717312 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:30.714789 5935 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 01 20:09:30.717312 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:30.714851 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s" podUID="e8bf3d0d-c668-44f1-9238-746e548faba0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 01 20:09:30.740316 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:30.739955 5935 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 01 20:09:30.740316 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:30.740026 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s" podUID="b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 01 20:09:30.763309 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:30.752455 5935 patch_prober.go:29] interesting pod/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 01 20:09:30.763309 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:30.752505 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd" podUID="8686280e-385b-4a98-a1d2-5018e5f08fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 01 20:09:30.763309 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:30.763053 5935 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 01 20:09:30.763309 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:30.763101 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k" podUID="7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 01 20:09:30.954328 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:30.953252 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/simulator-w4-0aa61b-kserve-6774594d4b-qmjz8" podUID="09414dc0-7742-48fa-a4b5-571199d99e15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:09:30.954328 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:30.953472 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0aa61b/simulator-w4-0aa61b-kserve-6774594d4b-qmjz8" Nov 01 20:09:31.271468 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:31.271416 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-w4-0aa61b-kserve-6674df959d-2lm95" podUID="b7f35703-1219-4dd4-8bbd-631fb7303724" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:09:31.272164 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:31.272119 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-e3c4a7/test-model-e3c4a7-kserve-59c98dffb5-cqbbk" podUID="b5df4b03-d387-44c7-a263-e24f9730ed0e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:09:31.272694 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:31.272678 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-e3c4a7/test-model-e3c4a7-kserve-59c98dffb5-cqbbk" Nov 01 20:09:31.361851 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:31.361821 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-e3c4a7/test-model-e3c4a7-kserve-59c98dffb5-cqbbk" Nov 01 20:09:31.744310 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:31.741430 5935 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 01 20:09:31.744310 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:31.741484 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s" podUID="e8bf3d0d-c668-44f1-9238-746e548faba0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 01 20:09:31.744310 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:31.741634 5935 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 01 20:09:31.744310 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:31.741676 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s" podUID="b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 01 20:09:31.768309 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:31.765563 5935 patch_prober.go:29] interesting pod/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 01 20:09:31.768309 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:31.765621 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd" podUID="8686280e-385b-4a98-a1d2-5018e5f08fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 01 20:09:31.768309 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:31.765887 5935 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 01 20:09:31.768309 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:31.765919 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k" podUID="7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 01 20:09:31.805605 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:31.803821 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0aa61b/simulator-w4-0aa61b-kserve-6774594d4b-qmjz8" Nov 01 20:09:32.732307 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:32.724661 5935 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 01 20:09:32.732307 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:32.724722 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s" podUID="e8bf3d0d-c668-44f1-9238-746e548faba0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 01 20:09:32.787565 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:32.786459 5935 patch_prober.go:29] interesting pod/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 01 20:09:32.787565 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:32.786524 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd" podUID="8686280e-385b-4a98-a1d2-5018e5f08fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 01 20:09:32.787565 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:32.787380 5935 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 01 20:09:32.787565 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:32.787425 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k" podUID="7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 01 20:09:32.787565 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:32.787494 5935 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 01 20:09:32.787565 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:32.787515 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s" podUID="b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 01 20:09:32.805115 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:32.803645 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5370bb-6c5d87878c-kng8d" Nov 01 20:09:32.806954 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:32.806225 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-5370bb-648c6878bc-j99qx" Nov 01 20:09:33.738360 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:33.719877 5935 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 01 20:09:33.738360 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:33.719943 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s" podUID="e8bf3d0d-c668-44f1-9238-746e548faba0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 01 20:09:33.752159 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:33.751727 5935 patch_prober.go:29] interesting pod/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 01 20:09:33.752159 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:33.751787 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd" podUID="8686280e-385b-4a98-a1d2-5018e5f08fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 01 20:09:33.758567 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:33.758538 5935 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 01 20:09:33.758729 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:33.758603 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s" podUID="b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 01 20:09:33.766578 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:33.765616 5935 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 01 20:09:33.766578 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:33.765662 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k" podUID="7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 01 20:09:34.713216 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:34.712779 5935 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Nov 01 20:09:34.713216 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:34.712854 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s" podUID="e8bf3d0d-c668-44f1-9238-746e548faba0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Nov 01 20:09:34.740689 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:34.740263 5935 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body= Nov 01 20:09:34.740689 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:34.740352 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s" podUID="b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" Nov 01 20:09:34.753190 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:34.752745 5935 patch_prober.go:29] interesting pod/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body= Nov 01 20:09:34.753190 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:34.752796 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd" podUID="8686280e-385b-4a98-a1d2-5018e5f08fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" Nov 01 20:09:34.761677 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:34.761573 5935 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Nov 01 20:09:34.761677 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:34.761621 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k" podUID="7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Nov 01 20:09:35.758471 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:35.758440 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd" Nov 01 20:09:35.759440 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:35.759411 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd" Nov 01 20:09:35.760937 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:35.760916 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd" Nov 01 20:09:35.766176 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:35.766150 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k" Nov 01 20:09:35.766617 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:35.766564 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k" Nov 01 20:09:35.767479 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:35.767449 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k" Nov 01 20:09:35.932172 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:09:35.932103 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:09:36.017866 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:36.017777 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s" Nov 01 20:09:36.019049 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:36.018901 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s" Nov 01 20:09:36.019611 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:36.019595 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s" Nov 01 20:09:36.140989 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:36.140961 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s" Nov 01 20:09:36.141302 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:36.141267 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s" Nov 01 20:09:36.142184 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:36.142164 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s" Nov 01 20:09:36.334681 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:36.334607 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5370bb-6f959ddc8c-l67b6" Nov 01 20:09:47.949471 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:47.949438 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-3a6480/distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f"] Nov 01 20:09:47.954912 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:47.954887 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3a6480/distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f" Nov 01 20:09:47.962896 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:47.962868 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-3a6480\"/\"default-dockercfg-98hrf\"" Nov 01 20:09:47.963038 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:47.962964 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-3a6480\"/\"kube-root-ca.crt\"" Nov 01 20:09:47.963121 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:47.963058 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-3a6480\"/\"openshift-service-ca.crt\"" Nov 01 20:09:47.964449 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:47.964427 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-3a6480\"/\"distinct-w0-3a6480-kserve-self-signed-certs\"" Nov 01 20:09:47.966677 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:47.966657 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-3a6480/distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f"] Nov 01 20:09:48.041823 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:48.041782 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e1d573b-e388-4d7b-b47a-8a01467c03b2-tls-certs\") pod \"distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f\" (UID: \"9e1d573b-e388-4d7b-b47a-8a01467c03b2\") " pod="e2e-models-e2e-worker-w0-3a6480/distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f" Nov 01 20:09:48.041998 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:48.041902 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e1d573b-e388-4d7b-b47a-8a01467c03b2-model-cache\") pod \"distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f\" (UID: \"9e1d573b-e388-4d7b-b47a-8a01467c03b2\") " pod="e2e-models-e2e-worker-w0-3a6480/distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f" Nov 01 20:09:48.041998 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:48.041942 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e1d573b-e388-4d7b-b47a-8a01467c03b2-dshm\") pod \"distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f\" (UID: \"9e1d573b-e388-4d7b-b47a-8a01467c03b2\") " pod="e2e-models-e2e-worker-w0-3a6480/distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f" Nov 01 20:09:48.042154 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:48.042131 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e1d573b-e388-4d7b-b47a-8a01467c03b2-home\") pod \"distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f\" (UID: \"9e1d573b-e388-4d7b-b47a-8a01467c03b2\") " pod="e2e-models-e2e-worker-w0-3a6480/distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f" Nov 01 20:09:48.042219 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:48.042188 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64tft\" (UniqueName: \"kubernetes.io/projected/9e1d573b-e388-4d7b-b47a-8a01467c03b2-kube-api-access-64tft\") pod \"distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f\" (UID: \"9e1d573b-e388-4d7b-b47a-8a01467c03b2\") " pod="e2e-models-e2e-worker-w0-3a6480/distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f" Nov 01 20:09:48.042272 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:48.042244 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e1d573b-e388-4d7b-b47a-8a01467c03b2-tmp-dir\") pod \"distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f\" (UID: \"9e1d573b-e388-4d7b-b47a-8a01467c03b2\") " pod="e2e-models-e2e-worker-w0-3a6480/distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f" Nov 01 20:09:48.143872 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:48.143118 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e1d573b-e388-4d7b-b47a-8a01467c03b2-home\") pod \"distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f\" (UID: \"9e1d573b-e388-4d7b-b47a-8a01467c03b2\") " pod="e2e-models-e2e-worker-w0-3a6480/distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f" Nov 01 20:09:48.143872 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:48.143172 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64tft\" (UniqueName: \"kubernetes.io/projected/9e1d573b-e388-4d7b-b47a-8a01467c03b2-kube-api-access-64tft\") pod \"distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f\" (UID: \"9e1d573b-e388-4d7b-b47a-8a01467c03b2\") " pod="e2e-models-e2e-worker-w0-3a6480/distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f" Nov 01 20:09:48.143872 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:48.143227 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e1d573b-e388-4d7b-b47a-8a01467c03b2-tmp-dir\") pod \"distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f\" (UID: \"9e1d573b-e388-4d7b-b47a-8a01467c03b2\") " pod="e2e-models-e2e-worker-w0-3a6480/distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f" Nov 01 20:09:48.143872 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:48.143254 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e1d573b-e388-4d7b-b47a-8a01467c03b2-tls-certs\") pod \"distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f\" (UID: \"9e1d573b-e388-4d7b-b47a-8a01467c03b2\") " pod="e2e-models-e2e-worker-w0-3a6480/distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f" Nov 01 20:09:48.143872 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:48.143340 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e1d573b-e388-4d7b-b47a-8a01467c03b2-model-cache\") pod \"distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f\" (UID: \"9e1d573b-e388-4d7b-b47a-8a01467c03b2\") " pod="e2e-models-e2e-worker-w0-3a6480/distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f" Nov 01 20:09:48.143872 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:48.143386 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e1d573b-e388-4d7b-b47a-8a01467c03b2-dshm\") pod \"distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f\" (UID: \"9e1d573b-e388-4d7b-b47a-8a01467c03b2\") " pod="e2e-models-e2e-worker-w0-3a6480/distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f" Nov 01 20:09:48.143872 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:48.143688 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e1d573b-e388-4d7b-b47a-8a01467c03b2-home\") pod \"distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f\" (UID: \"9e1d573b-e388-4d7b-b47a-8a01467c03b2\") " pod="e2e-models-e2e-worker-w0-3a6480/distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f" Nov 01 20:09:48.144383 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:48.144046 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e1d573b-e388-4d7b-b47a-8a01467c03b2-tmp-dir\") pod \"distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f\" (UID: \"9e1d573b-e388-4d7b-b47a-8a01467c03b2\") " pod="e2e-models-e2e-worker-w0-3a6480/distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f" Nov 01 20:09:48.144383 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:48.144210 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e1d573b-e388-4d7b-b47a-8a01467c03b2-model-cache\") pod \"distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f\" (UID: \"9e1d573b-e388-4d7b-b47a-8a01467c03b2\") " pod="e2e-models-e2e-worker-w0-3a6480/distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f" Nov 01 20:09:48.148215 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:48.148177 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e1d573b-e388-4d7b-b47a-8a01467c03b2-tls-certs\") pod \"distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f\" (UID: \"9e1d573b-e388-4d7b-b47a-8a01467c03b2\") " pod="e2e-models-e2e-worker-w0-3a6480/distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f" Nov 01 20:09:48.149730 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:48.149684 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e1d573b-e388-4d7b-b47a-8a01467c03b2-dshm\") pod \"distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f\" (UID: \"9e1d573b-e388-4d7b-b47a-8a01467c03b2\") " pod="e2e-models-e2e-worker-w0-3a6480/distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f" Nov 01 20:09:48.152014 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:48.151988 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64tft\" (UniqueName: \"kubernetes.io/projected/9e1d573b-e388-4d7b-b47a-8a01467c03b2-kube-api-access-64tft\") pod \"distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f\" (UID: \"9e1d573b-e388-4d7b-b47a-8a01467c03b2\") " pod="e2e-models-e2e-worker-w0-3a6480/distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f" Nov 01 20:09:48.269507 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:48.269434 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3a6480/distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f" Nov 01 20:09:48.547097 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:48.547011 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-3a6480/distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f"] Nov 01 20:09:48.554812 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:09:48.554770 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e1d573b_e388_4d7b_b47a_8a01467c03b2.slice/crio-534009b076a4ad0d2e4adb11fc1b1e340a3a726625288263ae6dc4fe68e293b3 WatchSource:0}: Error finding container 534009b076a4ad0d2e4adb11fc1b1e340a3a726625288263ae6dc4fe68e293b3: Status 404 returned error can't find the container with id 534009b076a4ad0d2e4adb11fc1b1e340a3a726625288263ae6dc4fe68e293b3 Nov 01 20:09:49.565990 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:49.565954 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3a6480/distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f" event={"ID":"9e1d573b-e388-4d7b-b47a-8a01467c03b2","Type":"ContainerStarted","Data":"6538548ecbbd4be1dc654fd71c83800b988ee6fb901db9c0a39e409e4b57843c"} Nov 01 20:09:49.565990 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:49.565993 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3a6480/distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f" event={"ID":"9e1d573b-e388-4d7b-b47a-8a01467c03b2","Type":"ContainerStarted","Data":"534009b076a4ad0d2e4adb11fc1b1e340a3a726625288263ae6dc4fe68e293b3"} Nov 01 20:09:49.582701 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:49.582657 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-3a6480/distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f" podStartSLOduration=2.582642804 podStartE2EDuration="2.582642804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:09:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:09:49.580604286 +0000 UTC m=+635.353764899" watchObservedRunningTime="2026-11-01 20:09:49.582642804 +0000 UTC m=+635.355803316" Nov 01 20:09:50.270074 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:50.270040 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3a6480/distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f" Nov 01 20:09:57.350124 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:57.350035 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-e3c4a7/test-model-e3c4a7-kserve-59c98dffb5-cqbbk"] Nov 01 20:09:58.270478 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:58.270440 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-3a6480/distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f" Nov 01 20:09:58.283468 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:58.283443 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-3a6480/distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f" Nov 01 20:09:58.646493 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:09:58.646425 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3a6480/distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f" Nov 01 20:10:05.844573 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:10:05.844533 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:10:12.377830 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:12.377774 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-e3c4a7/test-model-e3c4a7-kserve-59c98dffb5-cqbbk" podUID="b5df4b03-d387-44c7-a263-e24f9730ed0e" containerName="main" containerID="cri-o://ab4d91a5764bbfd36be6abe1c4f2f822b91d77ac2390181325a5618ec7a581ea" gracePeriod=45 Nov 01 20:10:12.634915 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:12.634859 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e3c4a7/test-model-e3c4a7-kserve-59c98dffb5-cqbbk" Nov 01 20:10:12.699158 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:12.699126 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5df4b03-d387-44c7-a263-e24f9730ed0e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5df4b03-d387-44c7-a263-e24f9730ed0e-dshm\") pod \"b5df4b03-d387-44c7-a263-e24f9730ed0e\" (UID: \"b5df4b03-d387-44c7-a263-e24f9730ed0e\") " Nov 01 20:10:12.699340 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:12.699325 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5df4b03-d387-44c7-a263-e24f9730ed0e-tls-certs\" (UniqueName: \"kubernetes.io/secret/b5df4b03-d387-44c7-a263-e24f9730ed0e-tls-certs\") pod \"b5df4b03-d387-44c7-a263-e24f9730ed0e\" (UID: \"b5df4b03-d387-44c7-a263-e24f9730ed0e\") " Nov 01 20:10:12.699391 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:12.699375 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5df4b03-d387-44c7-a263-e24f9730ed0e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5df4b03-d387-44c7-a263-e24f9730ed0e-model-cache\") pod \"b5df4b03-d387-44c7-a263-e24f9730ed0e\" (UID: \"b5df4b03-d387-44c7-a263-e24f9730ed0e\") " Nov 01 20:10:12.699468 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:12.699436 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5df4b03-d387-44c7-a263-e24f9730ed0e-kube-api-access-ksdfx\" (UniqueName: \"kubernetes.io/projected/b5df4b03-d387-44c7-a263-e24f9730ed0e-kube-api-access-ksdfx\") pod \"b5df4b03-d387-44c7-a263-e24f9730ed0e\" (UID: \"b5df4b03-d387-44c7-a263-e24f9730ed0e\") " Nov 01 20:10:12.699603 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:12.699479 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5df4b03-d387-44c7-a263-e24f9730ed0e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5df4b03-d387-44c7-a263-e24f9730ed0e-tmp-dir\") pod \"b5df4b03-d387-44c7-a263-e24f9730ed0e\" (UID: \"b5df4b03-d387-44c7-a263-e24f9730ed0e\") " Nov 01 20:10:12.699603 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:12.699533 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5df4b03-d387-44c7-a263-e24f9730ed0e-home\" (UniqueName: \"kubernetes.io/empty-dir/b5df4b03-d387-44c7-a263-e24f9730ed0e-home\") pod \"b5df4b03-d387-44c7-a263-e24f9730ed0e\" (UID: \"b5df4b03-d387-44c7-a263-e24f9730ed0e\") " Nov 01 20:10:12.699735 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:12.699708 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5df4b03-d387-44c7-a263-e24f9730ed0e-model-cache" pod "b5df4b03-d387-44c7-a263-e24f9730ed0e" (UID: "b5df4b03-d387-44c7-a263-e24f9730ed0e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:10:12.699848 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:12.699823 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5df4b03-d387-44c7-a263-e24f9730ed0e-tmp-dir" pod "b5df4b03-d387-44c7-a263-e24f9730ed0e" (UID: "b5df4b03-d387-44c7-a263-e24f9730ed0e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:10:12.699959 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:12.699939 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5df4b03-d387-44c7-a263-e24f9730ed0e-home" pod "b5df4b03-d387-44c7-a263-e24f9730ed0e" (UID: "b5df4b03-d387-44c7-a263-e24f9730ed0e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:10:12.700016 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:12.699995 5935 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5df4b03-d387-44c7-a263-e24f9730ed0e-model-cache\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:10:12.700075 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:12.700016 5935 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5df4b03-d387-44c7-a263-e24f9730ed0e-tmp-dir\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:10:12.701316 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:12.701275 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5df4b03-d387-44c7-a263-e24f9730ed0e-dshm" pod "b5df4b03-d387-44c7-a263-e24f9730ed0e" (UID: "b5df4b03-d387-44c7-a263-e24f9730ed0e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:10:12.701414 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:12.701386 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5df4b03-d387-44c7-a263-e24f9730ed0e-tls-certs" pod "b5df4b03-d387-44c7-a263-e24f9730ed0e" (UID: "b5df4b03-d387-44c7-a263-e24f9730ed0e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:10:12.701710 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:12.701690 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5df4b03-d387-44c7-a263-e24f9730ed0e-kube-api-access-ksdfx" pod "b5df4b03-d387-44c7-a263-e24f9730ed0e" (UID: "b5df4b03-d387-44c7-a263-e24f9730ed0e"). InnerVolumeSpecName "kube-api-access-ksdfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:10:12.801557 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:12.801536 5935 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5df4b03-d387-44c7-a263-e24f9730ed0e-tls-certs\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:10:12.801557 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:12.801556 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksdfx\" (UniqueName: \"kubernetes.io/projected/b5df4b03-d387-44c7-a263-e24f9730ed0e-kube-api-access-ksdfx\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:10:12.801713 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:12.801568 5935 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5df4b03-d387-44c7-a263-e24f9730ed0e-home\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:10:12.801713 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:12.801577 5935 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5df4b03-d387-44c7-a263-e24f9730ed0e-dshm\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:10:12.809187 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:12.809163 5935 generic.go:356] "Generic (PLEG): container finished" podID="b5df4b03-d387-44c7-a263-e24f9730ed0e" containerID="ab4d91a5764bbfd36be6abe1c4f2f822b91d77ac2390181325a5618ec7a581ea" exitCode=0 Nov 01 20:10:12.809324 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:12.809197 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e3c4a7/test-model-e3c4a7-kserve-59c98dffb5-cqbbk" event={"ID":"b5df4b03-d387-44c7-a263-e24f9730ed0e","Type":"ContainerDied","Data":"ab4d91a5764bbfd36be6abe1c4f2f822b91d77ac2390181325a5618ec7a581ea"} Nov 01 20:10:12.809324 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:12.809227 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-e3c4a7/test-model-e3c4a7-kserve-59c98dffb5-cqbbk" Nov 01 20:10:12.809324 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:12.809244 5935 scope.go:122] "RemoveContainer" containerID="ab4d91a5764bbfd36be6abe1c4f2f822b91d77ac2390181325a5618ec7a581ea" Nov 01 20:10:12.809484 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:12.809232 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-e3c4a7/test-model-e3c4a7-kserve-59c98dffb5-cqbbk" event={"ID":"b5df4b03-d387-44c7-a263-e24f9730ed0e","Type":"ContainerDied","Data":"ff44e0192dcdcb0b535dfe9401b96fbcc4e3b1b8fdd346baeb9265c2fb196946"} Nov 01 20:10:12.820425 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:12.820405 5935 scope.go:122] "RemoveContainer" containerID="ab4d91a5764bbfd36be6abe1c4f2f822b91d77ac2390181325a5618ec7a581ea" Nov 01 20:10:12.820698 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:10:12.820671 5935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab4d91a5764bbfd36be6abe1c4f2f822b91d77ac2390181325a5618ec7a581ea\": container with ID starting with ab4d91a5764bbfd36be6abe1c4f2f822b91d77ac2390181325a5618ec7a581ea not found: ID does not exist" containerID="ab4d91a5764bbfd36be6abe1c4f2f822b91d77ac2390181325a5618ec7a581ea" Nov 01 20:10:12.820773 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:12.820709 5935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab4d91a5764bbfd36be6abe1c4f2f822b91d77ac2390181325a5618ec7a581ea"} err="failed to get container status \"ab4d91a5764bbfd36be6abe1c4f2f822b91d77ac2390181325a5618ec7a581ea\": rpc error: code = NotFound desc = could not find container \"ab4d91a5764bbfd36be6abe1c4f2f822b91d77ac2390181325a5618ec7a581ea\": container with ID starting with ab4d91a5764bbfd36be6abe1c4f2f822b91d77ac2390181325a5618ec7a581ea not found: ID does not exist" Nov 01 20:10:12.834334 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:12.834311 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-e3c4a7/test-model-e3c4a7-kserve-59c98dffb5-cqbbk"] Nov 01 20:10:12.840584 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:12.840559 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-e3c4a7/test-model-e3c4a7-kserve-59c98dffb5-cqbbk"] Nov 01 20:10:13.469709 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:13.469680 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-e27ae7/distinct-w2-e27ae7-kserve-7df8547d56-9qgdk"] Nov 01 20:10:13.470212 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:13.470197 5935 cpu_manager.go:414] "containerMap: removing container" podUID="b5df4b03-d387-44c7-a263-e24f9730ed0e" containerName="main" Nov 01 20:10:13.470259 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:13.470215 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5df4b03-d387-44c7-a263-e24f9730ed0e" containerName="main" Nov 01 20:10:13.470383 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:13.470371 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5df4b03-d387-44c7-a263-e24f9730ed0e" containerName="main" Nov 01 20:10:13.477041 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:13.477013 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e27ae7/distinct-w2-e27ae7-kserve-7df8547d56-9qgdk" Nov 01 20:10:13.488214 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:13.488188 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e27ae7/distinct-w2-e27ae7-kserve-7df8547d56-9qgdk"] Nov 01 20:10:13.490496 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:13.490467 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-e27ae7\"/\"kube-root-ca.crt\"" Nov 01 20:10:13.490661 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:13.490495 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e27ae7\"/\"default-dockercfg-cqft8\"" Nov 01 20:10:13.490661 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:13.490609 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e27ae7\"/\"distinct-w2-e27ae7-kserve-self-signed-certs\"" Nov 01 20:10:13.490782 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:13.490697 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-e27ae7\"/\"openshift-service-ca.crt\"" Nov 01 20:10:13.616541 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:13.616495 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a948974-537a-49da-85e7-60adf68d1655-model-cache\") pod \"distinct-w2-e27ae7-kserve-7df8547d56-9qgdk\" (UID: \"4a948974-537a-49da-85e7-60adf68d1655\") " pod="e2e-models-e2e-worker-w2-e27ae7/distinct-w2-e27ae7-kserve-7df8547d56-9qgdk" Nov 01 20:10:13.616763 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:13.616604 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a948974-537a-49da-85e7-60adf68d1655-tmp-dir\") pod \"distinct-w2-e27ae7-kserve-7df8547d56-9qgdk\" (UID: \"4a948974-537a-49da-85e7-60adf68d1655\") " pod="e2e-models-e2e-worker-w2-e27ae7/distinct-w2-e27ae7-kserve-7df8547d56-9qgdk" Nov 01 20:10:13.616763 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:13.616648 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a948974-537a-49da-85e7-60adf68d1655-tls-certs\") pod \"distinct-w2-e27ae7-kserve-7df8547d56-9qgdk\" (UID: \"4a948974-537a-49da-85e7-60adf68d1655\") " pod="e2e-models-e2e-worker-w2-e27ae7/distinct-w2-e27ae7-kserve-7df8547d56-9qgdk" Nov 01 20:10:13.616884 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:13.616785 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqchf\" (UniqueName: \"kubernetes.io/projected/4a948974-537a-49da-85e7-60adf68d1655-kube-api-access-dqchf\") pod \"distinct-w2-e27ae7-kserve-7df8547d56-9qgdk\" (UID: \"4a948974-537a-49da-85e7-60adf68d1655\") " pod="e2e-models-e2e-worker-w2-e27ae7/distinct-w2-e27ae7-kserve-7df8547d56-9qgdk" Nov 01 20:10:13.616884 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:13.616830 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a948974-537a-49da-85e7-60adf68d1655-home\") pod \"distinct-w2-e27ae7-kserve-7df8547d56-9qgdk\" (UID: \"4a948974-537a-49da-85e7-60adf68d1655\") " pod="e2e-models-e2e-worker-w2-e27ae7/distinct-w2-e27ae7-kserve-7df8547d56-9qgdk" Nov 01 20:10:13.616884 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:13.616866 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a948974-537a-49da-85e7-60adf68d1655-dshm\") pod \"distinct-w2-e27ae7-kserve-7df8547d56-9qgdk\" (UID: \"4a948974-537a-49da-85e7-60adf68d1655\") " pod="e2e-models-e2e-worker-w2-e27ae7/distinct-w2-e27ae7-kserve-7df8547d56-9qgdk" Nov 01 20:10:13.718383 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:13.718346 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a948974-537a-49da-85e7-60adf68d1655-model-cache\") pod \"distinct-w2-e27ae7-kserve-7df8547d56-9qgdk\" (UID: \"4a948974-537a-49da-85e7-60adf68d1655\") " pod="e2e-models-e2e-worker-w2-e27ae7/distinct-w2-e27ae7-kserve-7df8547d56-9qgdk" Nov 01 20:10:13.718546 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:13.718434 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a948974-537a-49da-85e7-60adf68d1655-tmp-dir\") pod \"distinct-w2-e27ae7-kserve-7df8547d56-9qgdk\" (UID: \"4a948974-537a-49da-85e7-60adf68d1655\") " pod="e2e-models-e2e-worker-w2-e27ae7/distinct-w2-e27ae7-kserve-7df8547d56-9qgdk" Nov 01 20:10:13.718546 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:13.718473 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a948974-537a-49da-85e7-60adf68d1655-tls-certs\") pod \"distinct-w2-e27ae7-kserve-7df8547d56-9qgdk\" (UID: \"4a948974-537a-49da-85e7-60adf68d1655\") " pod="e2e-models-e2e-worker-w2-e27ae7/distinct-w2-e27ae7-kserve-7df8547d56-9qgdk" Nov 01 20:10:13.718681 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:13.718601 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqchf\" (UniqueName: \"kubernetes.io/projected/4a948974-537a-49da-85e7-60adf68d1655-kube-api-access-dqchf\") pod \"distinct-w2-e27ae7-kserve-7df8547d56-9qgdk\" (UID: \"4a948974-537a-49da-85e7-60adf68d1655\") " pod="e2e-models-e2e-worker-w2-e27ae7/distinct-w2-e27ae7-kserve-7df8547d56-9qgdk" Nov 01 20:10:13.718681 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:13.718647 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a948974-537a-49da-85e7-60adf68d1655-home\") pod \"distinct-w2-e27ae7-kserve-7df8547d56-9qgdk\" (UID: \"4a948974-537a-49da-85e7-60adf68d1655\") " pod="e2e-models-e2e-worker-w2-e27ae7/distinct-w2-e27ae7-kserve-7df8547d56-9qgdk" Nov 01 20:10:13.718793 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:13.718681 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a948974-537a-49da-85e7-60adf68d1655-dshm\") pod \"distinct-w2-e27ae7-kserve-7df8547d56-9qgdk\" (UID: \"4a948974-537a-49da-85e7-60adf68d1655\") " pod="e2e-models-e2e-worker-w2-e27ae7/distinct-w2-e27ae7-kserve-7df8547d56-9qgdk" Nov 01 20:10:13.718793 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:13.718705 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a948974-537a-49da-85e7-60adf68d1655-model-cache\") pod \"distinct-w2-e27ae7-kserve-7df8547d56-9qgdk\" (UID: \"4a948974-537a-49da-85e7-60adf68d1655\") " pod="e2e-models-e2e-worker-w2-e27ae7/distinct-w2-e27ae7-kserve-7df8547d56-9qgdk" Nov 01 20:10:13.718889 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:13.718791 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a948974-537a-49da-85e7-60adf68d1655-tmp-dir\") pod \"distinct-w2-e27ae7-kserve-7df8547d56-9qgdk\" (UID: \"4a948974-537a-49da-85e7-60adf68d1655\") " pod="e2e-models-e2e-worker-w2-e27ae7/distinct-w2-e27ae7-kserve-7df8547d56-9qgdk" Nov 01 20:10:13.719025 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:13.719003 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a948974-537a-49da-85e7-60adf68d1655-home\") pod \"distinct-w2-e27ae7-kserve-7df8547d56-9qgdk\" (UID: \"4a948974-537a-49da-85e7-60adf68d1655\") " pod="e2e-models-e2e-worker-w2-e27ae7/distinct-w2-e27ae7-kserve-7df8547d56-9qgdk" Nov 01 20:10:13.720744 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:13.720688 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a948974-537a-49da-85e7-60adf68d1655-dshm\") pod \"distinct-w2-e27ae7-kserve-7df8547d56-9qgdk\" (UID: \"4a948974-537a-49da-85e7-60adf68d1655\") " pod="e2e-models-e2e-worker-w2-e27ae7/distinct-w2-e27ae7-kserve-7df8547d56-9qgdk" Nov 01 20:10:13.720840 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:13.720768 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a948974-537a-49da-85e7-60adf68d1655-tls-certs\") pod \"distinct-w2-e27ae7-kserve-7df8547d56-9qgdk\" (UID: \"4a948974-537a-49da-85e7-60adf68d1655\") " pod="e2e-models-e2e-worker-w2-e27ae7/distinct-w2-e27ae7-kserve-7df8547d56-9qgdk" Nov 01 20:10:13.725937 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:13.725914 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqchf\" (UniqueName: \"kubernetes.io/projected/4a948974-537a-49da-85e7-60adf68d1655-kube-api-access-dqchf\") pod \"distinct-w2-e27ae7-kserve-7df8547d56-9qgdk\" (UID: \"4a948974-537a-49da-85e7-60adf68d1655\") " pod="e2e-models-e2e-worker-w2-e27ae7/distinct-w2-e27ae7-kserve-7df8547d56-9qgdk" Nov 01 20:10:13.796712 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:13.796659 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e27ae7/distinct-w2-e27ae7-kserve-7df8547d56-9qgdk" Nov 01 20:10:14.078324 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:14.078270 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e27ae7/distinct-w2-e27ae7-kserve-7df8547d56-9qgdk"] Nov 01 20:10:14.079904 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:10:14.079872 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4a948974_537a_49da_85e7_60adf68d1655.slice/crio-07380a329b12a2288e6484aca34c7ed601fd1a906a98b7948828a5ae6e495e75 WatchSource:0}: Error finding container 07380a329b12a2288e6484aca34c7ed601fd1a906a98b7948828a5ae6e495e75: Status 404 returned error can't find the container with id 07380a329b12a2288e6484aca34c7ed601fd1a906a98b7948828a5ae6e495e75 Nov 01 20:10:14.353430 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:14.353347 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5df4b03-d387-44c7-a263-e24f9730ed0e" path="/var/lib/kubelet/pods/b5df4b03-d387-44c7-a263-e24f9730ed0e/volumes" Nov 01 20:10:14.829963 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:14.829920 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e27ae7/distinct-w2-e27ae7-kserve-7df8547d56-9qgdk" event={"ID":"4a948974-537a-49da-85e7-60adf68d1655","Type":"ContainerStarted","Data":"cae5df9029be86bfb8b61d6230defafa4ae76e893262625fd835f376329609f6"} Nov 01 20:10:14.830729 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:14.830693 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e27ae7/distinct-w2-e27ae7-kserve-7df8547d56-9qgdk" event={"ID":"4a948974-537a-49da-85e7-60adf68d1655","Type":"ContainerStarted","Data":"07380a329b12a2288e6484aca34c7ed601fd1a906a98b7948828a5ae6e495e75"} Nov 01 20:10:15.798576 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:15.797978 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e27ae7/distinct-w2-e27ae7-kserve-7df8547d56-9qgdk" Nov 01 20:10:18.740481 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:18.740420 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-e27ae7/distinct-w2-e27ae7-kserve-7df8547d56-9qgdk" podStartSLOduration=5.740400919 podStartE2EDuration="5.740400919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:10:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:10:14.849449173 +0000 UTC m=+660.622609684" watchObservedRunningTime="2026-11-01 20:10:18.740400919 +0000 UTC m=+664.513561432" Nov 01 20:10:18.741447 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:18.741421 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e3c4a7-57c4dc9874-swj7w"] Nov 01 20:10:18.741724 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:18.741697 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e3c4a7-57c4dc9874-swj7w" podUID="2f7b4b6e-9a7c-4ce8-aa0b-600cded35eb3" containerName="maas-api" containerID="cri-o://e83d858f53c7d1c0013d1e6c9fb0aefcea45bb727c7700354c116460b3c23ee5" gracePeriod=30 Nov 01 20:10:19.300488 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:19.300425 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e3c4a7-57c4dc9874-swj7w" Nov 01 20:10:19.369196 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:19.369166 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f7b4b6e-9a7c-4ce8-aa0b-600cded35eb3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f7b4b6e-9a7c-4ce8-aa0b-600cded35eb3-metrics-certs\") pod \"2f7b4b6e-9a7c-4ce8-aa0b-600cded35eb3\" (UID: \"2f7b4b6e-9a7c-4ce8-aa0b-600cded35eb3\") " Nov 01 20:10:19.369399 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:19.369235 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f7b4b6e-9a7c-4ce8-aa0b-600cded35eb3-kube-api-access-wtbdw\" (UniqueName: \"kubernetes.io/projected/2f7b4b6e-9a7c-4ce8-aa0b-600cded35eb3-kube-api-access-wtbdw\") pod \"2f7b4b6e-9a7c-4ce8-aa0b-600cded35eb3\" (UID: \"2f7b4b6e-9a7c-4ce8-aa0b-600cded35eb3\") " Nov 01 20:10:19.369399 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:19.369263 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f7b4b6e-9a7c-4ce8-aa0b-600cded35eb3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f7b4b6e-9a7c-4ce8-aa0b-600cded35eb3-maas-api-tls\") pod \"2f7b4b6e-9a7c-4ce8-aa0b-600cded35eb3\" (UID: \"2f7b4b6e-9a7c-4ce8-aa0b-600cded35eb3\") " Nov 01 20:10:19.369399 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:19.369354 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f7b4b6e-9a7c-4ce8-aa0b-600cded35eb3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f7b4b6e-9a7c-4ce8-aa0b-600cded35eb3-openshift-service-ca\") pod \"2f7b4b6e-9a7c-4ce8-aa0b-600cded35eb3\" (UID: \"2f7b4b6e-9a7c-4ce8-aa0b-600cded35eb3\") " Nov 01 20:10:19.369769 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:19.369738 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f7b4b6e-9a7c-4ce8-aa0b-600cded35eb3-openshift-service-ca" pod "2f7b4b6e-9a7c-4ce8-aa0b-600cded35eb3" (UID: "2f7b4b6e-9a7c-4ce8-aa0b-600cded35eb3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:10:19.371514 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:19.371486 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f7b4b6e-9a7c-4ce8-aa0b-600cded35eb3-kube-api-access-wtbdw" pod "2f7b4b6e-9a7c-4ce8-aa0b-600cded35eb3" (UID: "2f7b4b6e-9a7c-4ce8-aa0b-600cded35eb3"). InnerVolumeSpecName "kube-api-access-wtbdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:10:19.371514 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:19.371490 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f7b4b6e-9a7c-4ce8-aa0b-600cded35eb3-maas-api-tls" pod "2f7b4b6e-9a7c-4ce8-aa0b-600cded35eb3" (UID: "2f7b4b6e-9a7c-4ce8-aa0b-600cded35eb3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:10:19.371678 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:19.371589 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f7b4b6e-9a7c-4ce8-aa0b-600cded35eb3-metrics-certs" pod "2f7b4b6e-9a7c-4ce8-aa0b-600cded35eb3" (UID: "2f7b4b6e-9a7c-4ce8-aa0b-600cded35eb3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:10:19.471238 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:19.471208 5935 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2f7b4b6e-9a7c-4ce8-aa0b-600cded35eb3-metrics-certs\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:10:19.471238 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:19.471235 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtbdw\" (UniqueName: \"kubernetes.io/projected/2f7b4b6e-9a7c-4ce8-aa0b-600cded35eb3-kube-api-access-wtbdw\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:10:19.471238 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:19.471245 5935 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2f7b4b6e-9a7c-4ce8-aa0b-600cded35eb3-maas-api-tls\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:10:19.471498 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:19.471254 5935 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f7b4b6e-9a7c-4ce8-aa0b-600cded35eb3-openshift-service-ca\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:10:19.879852 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:19.879817 5935 generic.go:356] "Generic (PLEG): container finished" podID="2f7b4b6e-9a7c-4ce8-aa0b-600cded35eb3" containerID="e83d858f53c7d1c0013d1e6c9fb0aefcea45bb727c7700354c116460b3c23ee5" exitCode=0 Nov 01 20:10:19.880261 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:19.879865 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e3c4a7-57c4dc9874-swj7w" Nov 01 20:10:19.880261 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:19.879911 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e3c4a7-57c4dc9874-swj7w" event={"ID":"2f7b4b6e-9a7c-4ce8-aa0b-600cded35eb3","Type":"ContainerDied","Data":"e83d858f53c7d1c0013d1e6c9fb0aefcea45bb727c7700354c116460b3c23ee5"} Nov 01 20:10:19.880261 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:19.879965 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e3c4a7-57c4dc9874-swj7w" event={"ID":"2f7b4b6e-9a7c-4ce8-aa0b-600cded35eb3","Type":"ContainerDied","Data":"a7ebe7d20825a298517584f1004db817ac9e14c7b487ce384bc64e0cbb545615"} Nov 01 20:10:19.880261 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:19.879987 5935 scope.go:122] "RemoveContainer" containerID="e83d858f53c7d1c0013d1e6c9fb0aefcea45bb727c7700354c116460b3c23ee5" Nov 01 20:10:19.890028 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:19.890011 5935 scope.go:122] "RemoveContainer" containerID="e83d858f53c7d1c0013d1e6c9fb0aefcea45bb727c7700354c116460b3c23ee5" Nov 01 20:10:19.890306 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:10:19.890270 5935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e83d858f53c7d1c0013d1e6c9fb0aefcea45bb727c7700354c116460b3c23ee5\": container with ID starting with e83d858f53c7d1c0013d1e6c9fb0aefcea45bb727c7700354c116460b3c23ee5 not found: ID does not exist" containerID="e83d858f53c7d1c0013d1e6c9fb0aefcea45bb727c7700354c116460b3c23ee5" Nov 01 20:10:19.890396 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:19.890315 5935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e83d858f53c7d1c0013d1e6c9fb0aefcea45bb727c7700354c116460b3c23ee5"} err="failed to get container status \"e83d858f53c7d1c0013d1e6c9fb0aefcea45bb727c7700354c116460b3c23ee5\": rpc error: code = NotFound desc = could not find container \"e83d858f53c7d1c0013d1e6c9fb0aefcea45bb727c7700354c116460b3c23ee5\": container with ID starting with e83d858f53c7d1c0013d1e6c9fb0aefcea45bb727c7700354c116460b3c23ee5 not found: ID does not exist" Nov 01 20:10:19.901728 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:19.901705 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e3c4a7-57c4dc9874-swj7w"] Nov 01 20:10:19.905293 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:19.905252 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e3c4a7-57c4dc9874-swj7w"] Nov 01 20:10:20.356371 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:20.355302 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f7b4b6e-9a7c-4ce8-aa0b-600cded35eb3" path="/var/lib/kubelet/pods/2f7b4b6e-9a7c-4ce8-aa0b-600cded35eb3/volumes" Nov 01 20:10:23.797153 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:23.797118 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-e27ae7/distinct-w2-e27ae7-kserve-7df8547d56-9qgdk" Nov 01 20:10:23.810136 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:23.810114 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-e27ae7/distinct-w2-e27ae7-kserve-7df8547d56-9qgdk" Nov 01 20:10:23.928260 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:23.928234 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e27ae7/distinct-w2-e27ae7-kserve-7df8547d56-9qgdk" Nov 01 20:10:26.248533 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:26.248499 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-e27ae7/distinct-2-w2-e27ae7-kserve-6bf4d46475-jdjhp"] Nov 01 20:10:26.249023 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:26.249009 5935 cpu_manager.go:414] "containerMap: removing container" podUID="2f7b4b6e-9a7c-4ce8-aa0b-600cded35eb3" containerName="maas-api" Nov 01 20:10:26.249023 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:26.249023 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f7b4b6e-9a7c-4ce8-aa0b-600cded35eb3" containerName="maas-api" Nov 01 20:10:26.249154 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:26.249110 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f7b4b6e-9a7c-4ce8-aa0b-600cded35eb3" containerName="maas-api" Nov 01 20:10:26.254331 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:26.254309 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e27ae7/distinct-2-w2-e27ae7-kserve-6bf4d46475-jdjhp" Nov 01 20:10:26.262353 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:26.262310 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e27ae7\"/\"distinct-2-w2-e27ae7-kserve-self-signed-certs\"" Nov 01 20:10:26.265638 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:26.265613 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e27ae7/distinct-2-w2-e27ae7-kserve-6bf4d46475-jdjhp"] Nov 01 20:10:26.349833 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:26.349787 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b916d612-a355-4525-8143-9e3a47ead6d0-tmp-dir\") pod \"distinct-2-w2-e27ae7-kserve-6bf4d46475-jdjhp\" (UID: \"b916d612-a355-4525-8143-9e3a47ead6d0\") " pod="e2e-models-e2e-worker-w2-e27ae7/distinct-2-w2-e27ae7-kserve-6bf4d46475-jdjhp" Nov 01 20:10:26.350139 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:26.350111 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b916d612-a355-4525-8143-9e3a47ead6d0-model-cache\") pod \"distinct-2-w2-e27ae7-kserve-6bf4d46475-jdjhp\" (UID: \"b916d612-a355-4525-8143-9e3a47ead6d0\") " pod="e2e-models-e2e-worker-w2-e27ae7/distinct-2-w2-e27ae7-kserve-6bf4d46475-jdjhp" Nov 01 20:10:26.350343 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:26.350160 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b916d612-a355-4525-8143-9e3a47ead6d0-home\") pod \"distinct-2-w2-e27ae7-kserve-6bf4d46475-jdjhp\" (UID: \"b916d612-a355-4525-8143-9e3a47ead6d0\") " pod="e2e-models-e2e-worker-w2-e27ae7/distinct-2-w2-e27ae7-kserve-6bf4d46475-jdjhp" Nov 01 20:10:26.350343 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:26.350216 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b916d612-a355-4525-8143-9e3a47ead6d0-tls-certs\") pod \"distinct-2-w2-e27ae7-kserve-6bf4d46475-jdjhp\" (UID: \"b916d612-a355-4525-8143-9e3a47ead6d0\") " pod="e2e-models-e2e-worker-w2-e27ae7/distinct-2-w2-e27ae7-kserve-6bf4d46475-jdjhp" Nov 01 20:10:26.350343 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:26.350251 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b916d612-a355-4525-8143-9e3a47ead6d0-dshm\") pod \"distinct-2-w2-e27ae7-kserve-6bf4d46475-jdjhp\" (UID: \"b916d612-a355-4525-8143-9e3a47ead6d0\") " pod="e2e-models-e2e-worker-w2-e27ae7/distinct-2-w2-e27ae7-kserve-6bf4d46475-jdjhp" Nov 01 20:10:26.350343 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:26.350278 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv6lk\" (UniqueName: \"kubernetes.io/projected/b916d612-a355-4525-8143-9e3a47ead6d0-kube-api-access-kv6lk\") pod \"distinct-2-w2-e27ae7-kserve-6bf4d46475-jdjhp\" (UID: \"b916d612-a355-4525-8143-9e3a47ead6d0\") " pod="e2e-models-e2e-worker-w2-e27ae7/distinct-2-w2-e27ae7-kserve-6bf4d46475-jdjhp" Nov 01 20:10:26.451821 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:26.451783 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b916d612-a355-4525-8143-9e3a47ead6d0-tmp-dir\") pod \"distinct-2-w2-e27ae7-kserve-6bf4d46475-jdjhp\" (UID: \"b916d612-a355-4525-8143-9e3a47ead6d0\") " pod="e2e-models-e2e-worker-w2-e27ae7/distinct-2-w2-e27ae7-kserve-6bf4d46475-jdjhp" Nov 01 20:10:26.451989 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:26.451862 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b916d612-a355-4525-8143-9e3a47ead6d0-model-cache\") pod \"distinct-2-w2-e27ae7-kserve-6bf4d46475-jdjhp\" (UID: \"b916d612-a355-4525-8143-9e3a47ead6d0\") " pod="e2e-models-e2e-worker-w2-e27ae7/distinct-2-w2-e27ae7-kserve-6bf4d46475-jdjhp" Nov 01 20:10:26.451989 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:26.451905 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b916d612-a355-4525-8143-9e3a47ead6d0-home\") pod \"distinct-2-w2-e27ae7-kserve-6bf4d46475-jdjhp\" (UID: \"b916d612-a355-4525-8143-9e3a47ead6d0\") " pod="e2e-models-e2e-worker-w2-e27ae7/distinct-2-w2-e27ae7-kserve-6bf4d46475-jdjhp" Nov 01 20:10:26.451989 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:26.451968 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b916d612-a355-4525-8143-9e3a47ead6d0-tls-certs\") pod \"distinct-2-w2-e27ae7-kserve-6bf4d46475-jdjhp\" (UID: \"b916d612-a355-4525-8143-9e3a47ead6d0\") " pod="e2e-models-e2e-worker-w2-e27ae7/distinct-2-w2-e27ae7-kserve-6bf4d46475-jdjhp" Nov 01 20:10:26.452089 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:26.452000 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b916d612-a355-4525-8143-9e3a47ead6d0-dshm\") pod \"distinct-2-w2-e27ae7-kserve-6bf4d46475-jdjhp\" (UID: \"b916d612-a355-4525-8143-9e3a47ead6d0\") " pod="e2e-models-e2e-worker-w2-e27ae7/distinct-2-w2-e27ae7-kserve-6bf4d46475-jdjhp" Nov 01 20:10:26.452089 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:26.452025 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv6lk\" (UniqueName: \"kubernetes.io/projected/b916d612-a355-4525-8143-9e3a47ead6d0-kube-api-access-kv6lk\") pod \"distinct-2-w2-e27ae7-kserve-6bf4d46475-jdjhp\" (UID: \"b916d612-a355-4525-8143-9e3a47ead6d0\") " pod="e2e-models-e2e-worker-w2-e27ae7/distinct-2-w2-e27ae7-kserve-6bf4d46475-jdjhp" Nov 01 20:10:26.452710 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:26.452681 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b916d612-a355-4525-8143-9e3a47ead6d0-home\") pod \"distinct-2-w2-e27ae7-kserve-6bf4d46475-jdjhp\" (UID: \"b916d612-a355-4525-8143-9e3a47ead6d0\") " pod="e2e-models-e2e-worker-w2-e27ae7/distinct-2-w2-e27ae7-kserve-6bf4d46475-jdjhp" Nov 01 20:10:26.452898 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:26.452872 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b916d612-a355-4525-8143-9e3a47ead6d0-tmp-dir\") pod \"distinct-2-w2-e27ae7-kserve-6bf4d46475-jdjhp\" (UID: \"b916d612-a355-4525-8143-9e3a47ead6d0\") " pod="e2e-models-e2e-worker-w2-e27ae7/distinct-2-w2-e27ae7-kserve-6bf4d46475-jdjhp" Nov 01 20:10:26.453010 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:26.452930 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b916d612-a355-4525-8143-9e3a47ead6d0-model-cache\") pod \"distinct-2-w2-e27ae7-kserve-6bf4d46475-jdjhp\" (UID: \"b916d612-a355-4525-8143-9e3a47ead6d0\") " pod="e2e-models-e2e-worker-w2-e27ae7/distinct-2-w2-e27ae7-kserve-6bf4d46475-jdjhp" Nov 01 20:10:26.470055 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:26.468979 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b916d612-a355-4525-8143-9e3a47ead6d0-tls-certs\") pod \"distinct-2-w2-e27ae7-kserve-6bf4d46475-jdjhp\" (UID: \"b916d612-a355-4525-8143-9e3a47ead6d0\") " pod="e2e-models-e2e-worker-w2-e27ae7/distinct-2-w2-e27ae7-kserve-6bf4d46475-jdjhp" Nov 01 20:10:26.474310 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:26.472560 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b916d612-a355-4525-8143-9e3a47ead6d0-dshm\") pod \"distinct-2-w2-e27ae7-kserve-6bf4d46475-jdjhp\" (UID: \"b916d612-a355-4525-8143-9e3a47ead6d0\") " pod="e2e-models-e2e-worker-w2-e27ae7/distinct-2-w2-e27ae7-kserve-6bf4d46475-jdjhp" Nov 01 20:10:26.477002 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:26.476976 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv6lk\" (UniqueName: \"kubernetes.io/projected/b916d612-a355-4525-8143-9e3a47ead6d0-kube-api-access-kv6lk\") pod \"distinct-2-w2-e27ae7-kserve-6bf4d46475-jdjhp\" (UID: \"b916d612-a355-4525-8143-9e3a47ead6d0\") " pod="e2e-models-e2e-worker-w2-e27ae7/distinct-2-w2-e27ae7-kserve-6bf4d46475-jdjhp" Nov 01 20:10:26.571083 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:26.571010 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e27ae7/distinct-2-w2-e27ae7-kserve-6bf4d46475-jdjhp" Nov 01 20:10:26.851864 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:26.851776 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e27ae7/distinct-2-w2-e27ae7-kserve-6bf4d46475-jdjhp"] Nov 01 20:10:26.858138 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:10:26.858109 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb916d612_a355_4525_8143_9e3a47ead6d0.slice/crio-28472daaab66c58d9f340e63390add390fe32fd610d058f0ee56e64c3366dc76 WatchSource:0}: Error finding container 28472daaab66c58d9f340e63390add390fe32fd610d058f0ee56e64c3366dc76: Status 404 returned error can't find the container with id 28472daaab66c58d9f340e63390add390fe32fd610d058f0ee56e64c3366dc76 Nov 01 20:10:26.939244 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:26.939217 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e27ae7/distinct-2-w2-e27ae7-kserve-6bf4d46475-jdjhp" event={"ID":"b916d612-a355-4525-8143-9e3a47ead6d0","Type":"ContainerStarted","Data":"0ca1f24ac8ccb1cf5520880d24e4719caf99cc6c1de37d73d8f2239df2d2190d"} Nov 01 20:10:26.939367 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:26.939264 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e27ae7/distinct-2-w2-e27ae7-kserve-6bf4d46475-jdjhp" event={"ID":"b916d612-a355-4525-8143-9e3a47ead6d0","Type":"ContainerStarted","Data":"28472daaab66c58d9f340e63390add390fe32fd610d058f0ee56e64c3366dc76"} Nov 01 20:10:26.955944 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:26.955901 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-e27ae7/distinct-2-w2-e27ae7-kserve-6bf4d46475-jdjhp" podStartSLOduration=0.955887744 podStartE2EDuration="955.887744ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:10:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:10:26.954784796 +0000 UTC m=+672.727945311" watchObservedRunningTime="2026-11-01 20:10:26.955887744 +0000 UTC m=+672.729048256" Nov 01 20:10:27.571437 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:27.571405 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e27ae7/distinct-2-w2-e27ae7-kserve-6bf4d46475-jdjhp" Nov 01 20:10:29.639416 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:29.639365 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f" podUID="9e1d573b-e388-4d7b-b47a-8a01467c03b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:10:29.923526 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:29.923414 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e27ae7/distinct-w2-e27ae7-kserve-7df8547d56-9qgdk" podUID="4a948974-537a-49da-85e7-60adf68d1655" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:10:30.227314 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:30.226393 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-w4-0aa61b-kserve-6674df959d-2lm95" podUID="b7f35703-1219-4dd4-8bbd-631fb7303724" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 20:10:30.822319 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:30.819365 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/simulator-w4-0aa61b-kserve-6774594d4b-qmjz8" podUID="09414dc0-7742-48fa-a4b5-571199d99e15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:10:35.844332 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:10:35.844263 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:10:36.572258 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:36.572228 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-e27ae7/distinct-2-w2-e27ae7-kserve-6bf4d46475-jdjhp" Nov 01 20:10:36.585552 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:36.585524 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-e27ae7/distinct-2-w2-e27ae7-kserve-6bf4d46475-jdjhp" Nov 01 20:10:37.284404 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:37.284375 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e27ae7/distinct-2-w2-e27ae7-kserve-6bf4d46475-jdjhp" Nov 01 20:10:44.237309 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:44.233532 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-w4-0aa61b-kserve-6674df959d-2lm95" podUID="b7f35703-1219-4dd4-8bbd-631fb7303724" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 20:10:44.642480 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:44.642377 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f" podUID="9e1d573b-e388-4d7b-b47a-8a01467c03b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:10:44.889439 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:44.889398 5935 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e27ae7/distinct-w2-e27ae7-kserve-7df8547d56-9qgdk" podUID="4a948974-537a-49da-85e7-60adf68d1655" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": context deadline exceeded" Nov 01 20:10:44.902436 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:44.902347 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/simulator-w4-0aa61b-kserve-6774594d4b-qmjz8" podUID="09414dc0-7742-48fa-a4b5-571199d99e15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:10:44.940305 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:44.940241 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e27ae7/distinct-w2-e27ae7-kserve-7df8547d56-9qgdk" podUID="4a948974-537a-49da-85e7-60adf68d1655" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:10:45.236407 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:45.236360 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-w4-0aa61b-kserve-6674df959d-2lm95" podUID="b7f35703-1219-4dd4-8bbd-631fb7303724" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": context deadline exceeded" Nov 01 20:10:45.237107 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:45.237085 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-w4-0aa61b-kserve-6674df959d-2lm95" Nov 01 20:10:45.277174 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:45.277135 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e27ae7/distinct-2-w2-e27ae7-kserve-6bf4d46475-jdjhp" podUID="b916d612-a355-4525-8143-9e3a47ead6d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:10:45.527003 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:45.526916 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-w4-0aa61b-kserve-6674df959d-2lm95" Nov 01 20:10:45.689312 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:45.665766 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f" podUID="9e1d573b-e388-4d7b-b47a-8a01467c03b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 01 20:10:45.781395 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:45.781267 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3a6480/distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f" Nov 01 20:10:45.944422 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:45.944370 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e27ae7/distinct-w2-e27ae7-kserve-7df8547d56-9qgdk" podUID="4a948974-537a-49da-85e7-60adf68d1655" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:10:45.948310 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:45.946011 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e27ae7/distinct-w2-e27ae7-kserve-7df8547d56-9qgdk" Nov 01 20:10:46.158186 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:46.158100 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e27ae7/distinct-w2-e27ae7-kserve-7df8547d56-9qgdk" Nov 01 20:10:46.186211 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:46.186180 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3a6480/distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f" Nov 01 20:10:59.160501 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:59.160437 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e27ae7/distinct-w2-e27ae7-kserve-7df8547d56-9qgdk" podUID="4a948974-537a-49da-85e7-60adf68d1655" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:10:59.189403 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:59.189355 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f" podUID="9e1d573b-e388-4d7b-b47a-8a01467c03b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:10:59.283762 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:59.283663 5935 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-3a6480/distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f" podUID="9e1d573b-e388-4d7b-b47a-8a01467c03b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:10:59.546173 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:59.545372 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-w4-0aa61b-kserve-6674df959d-2lm95" podUID="b7f35703-1219-4dd4-8bbd-631fb7303724" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:10:59.991320 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:10:59.990615 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/simulator-w4-0aa61b-kserve-6774594d4b-qmjz8" podUID="09414dc0-7742-48fa-a4b5-571199d99e15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:11:00.248363 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:00.247476 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f" podUID="9e1d573b-e388-4d7b-b47a-8a01467c03b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 01 20:11:00.280315 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:00.248993 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3a6480/distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f" Nov 01 20:11:00.480120 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:00.480089 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3a6480/distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f" Nov 01 20:11:05.849256 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:11:05.849212 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:11:15.353856 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:15.353810 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-7f0c86-946796b95-bbvrc"] Nov 01 20:11:15.359012 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:15.358992 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7f0c86-946796b95-bbvrc" Nov 01 20:11:15.364109 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:15.364089 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-7f0c86\"" Nov 01 20:11:15.364358 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:15.364333 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-7f0c86-dockercfg-tlj7c\"" Nov 01 20:11:15.368078 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:15.368054 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-7f0c86-946796b95-bbvrc"] Nov 01 20:11:15.399583 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:15.399547 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-7f0c86-84cd8cd7d9-pfqdn"] Nov 01 20:11:15.403564 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:15.403543 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-7f0c86-84cd8cd7d9-pfqdn" Nov 01 20:11:15.415416 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:15.415392 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-7f0c86-84cd8cd7d9-pfqdn"] Nov 01 20:11:15.502184 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:15.502129 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/657cdb6c-3591-4340-8737-c15687f3f32b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-7f0c86-946796b95-bbvrc\" (UID: \"657cdb6c-3591-4340-8737-c15687f3f32b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7f0c86-946796b95-bbvrc" Nov 01 20:11:15.502388 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:15.502212 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3b6aa7b-cfb2-48b4-a762-e3dd5cbb1f41-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-7f0c86-84cd8cd7d9-pfqdn\" (UID: \"e3b6aa7b-cfb2-48b4-a762-e3dd5cbb1f41\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-7f0c86-84cd8cd7d9-pfqdn" Nov 01 20:11:15.502531 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:15.502508 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b49b\" (UniqueName: \"kubernetes.io/projected/657cdb6c-3591-4340-8737-c15687f3f32b-kube-api-access-7b49b\") pod \"payload-pre-processing-e2e-praxis-b-7f0c86-946796b95-bbvrc\" (UID: \"657cdb6c-3591-4340-8737-c15687f3f32b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7f0c86-946796b95-bbvrc" Nov 01 20:11:15.502599 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:15.502546 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqbmv\" (UniqueName: \"kubernetes.io/projected/e3b6aa7b-cfb2-48b4-a762-e3dd5cbb1f41-kube-api-access-fqbmv\") pod \"payload-processing-e2e-praxis-b-7f0c86-84cd8cd7d9-pfqdn\" (UID: \"e3b6aa7b-cfb2-48b4-a762-e3dd5cbb1f41\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-7f0c86-84cd8cd7d9-pfqdn" Nov 01 20:11:15.603693 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:15.603654 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b49b\" (UniqueName: \"kubernetes.io/projected/657cdb6c-3591-4340-8737-c15687f3f32b-kube-api-access-7b49b\") pod \"payload-pre-processing-e2e-praxis-b-7f0c86-946796b95-bbvrc\" (UID: \"657cdb6c-3591-4340-8737-c15687f3f32b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7f0c86-946796b95-bbvrc" Nov 01 20:11:15.603693 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:15.603688 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqbmv\" (UniqueName: \"kubernetes.io/projected/e3b6aa7b-cfb2-48b4-a762-e3dd5cbb1f41-kube-api-access-fqbmv\") pod \"payload-processing-e2e-praxis-b-7f0c86-84cd8cd7d9-pfqdn\" (UID: \"e3b6aa7b-cfb2-48b4-a762-e3dd5cbb1f41\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-7f0c86-84cd8cd7d9-pfqdn" Nov 01 20:11:15.603973 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:15.603728 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/657cdb6c-3591-4340-8737-c15687f3f32b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-7f0c86-946796b95-bbvrc\" (UID: \"657cdb6c-3591-4340-8737-c15687f3f32b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7f0c86-946796b95-bbvrc" Nov 01 20:11:15.603973 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:15.603747 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3b6aa7b-cfb2-48b4-a762-e3dd5cbb1f41-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-7f0c86-84cd8cd7d9-pfqdn\" (UID: \"e3b6aa7b-cfb2-48b4-a762-e3dd5cbb1f41\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-7f0c86-84cd8cd7d9-pfqdn" Nov 01 20:11:15.604537 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:15.604511 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/657cdb6c-3591-4340-8737-c15687f3f32b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-7f0c86-946796b95-bbvrc\" (UID: \"657cdb6c-3591-4340-8737-c15687f3f32b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7f0c86-946796b95-bbvrc" Nov 01 20:11:15.604596 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:15.604533 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3b6aa7b-cfb2-48b4-a762-e3dd5cbb1f41-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-7f0c86-84cd8cd7d9-pfqdn\" (UID: \"e3b6aa7b-cfb2-48b4-a762-e3dd5cbb1f41\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-7f0c86-84cd8cd7d9-pfqdn" Nov 01 20:11:15.611912 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:15.611889 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b49b\" (UniqueName: \"kubernetes.io/projected/657cdb6c-3591-4340-8737-c15687f3f32b-kube-api-access-7b49b\") pod \"payload-pre-processing-e2e-praxis-b-7f0c86-946796b95-bbvrc\" (UID: \"657cdb6c-3591-4340-8737-c15687f3f32b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7f0c86-946796b95-bbvrc" Nov 01 20:11:15.612023 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:15.611936 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqbmv\" (UniqueName: \"kubernetes.io/projected/e3b6aa7b-cfb2-48b4-a762-e3dd5cbb1f41-kube-api-access-fqbmv\") pod \"payload-processing-e2e-praxis-b-7f0c86-84cd8cd7d9-pfqdn\" (UID: \"e3b6aa7b-cfb2-48b4-a762-e3dd5cbb1f41\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-7f0c86-84cd8cd7d9-pfqdn" Nov 01 20:11:15.678814 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:15.678782 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7f0c86-946796b95-bbvrc" Nov 01 20:11:15.719891 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:15.719863 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-7f0c86-84cd8cd7d9-pfqdn" Nov 01 20:11:15.962133 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:15.962107 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-7f0c86-946796b95-bbvrc"] Nov 01 20:11:15.963331 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:11:15.963307 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod657cdb6c_3591_4340_8737_c15687f3f32b.slice/crio-63c36b375587bd3aea8ca3113751b84561141d5b9fdaecb9357aa1ea6e8acc43 WatchSource:0}: Error finding container 63c36b375587bd3aea8ca3113751b84561141d5b9fdaecb9357aa1ea6e8acc43: Status 404 returned error can't find the container with id 63c36b375587bd3aea8ca3113751b84561141d5b9fdaecb9357aa1ea6e8acc43 Nov 01 20:11:16.021517 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:16.021488 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-7f0c86-84cd8cd7d9-pfqdn"] Nov 01 20:11:16.023459 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:11:16.023434 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3b6aa7b_cfb2_48b4_a762_e3dd5cbb1f41.slice/crio-86b4063c01e650fcdd44cd57441f70aa86201b55b2d446ca8fe833e4e23fffad WatchSource:0}: Error finding container 86b4063c01e650fcdd44cd57441f70aa86201b55b2d446ca8fe833e4e23fffad: Status 404 returned error can't find the container with id 86b4063c01e650fcdd44cd57441f70aa86201b55b2d446ca8fe833e4e23fffad Nov 01 20:11:16.232277 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:16.232191 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-7f0c86-84cd8cd7d9-pfqdn" event={"ID":"e3b6aa7b-cfb2-48b4-a762-e3dd5cbb1f41","Type":"ContainerStarted","Data":"e065ac96545bda94bbc74205a70d2b8f1e2bcfe9ab9fce04ab6148f67019c389"} Nov 01 20:11:16.232277 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:16.232254 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-7f0c86-84cd8cd7d9-pfqdn" event={"ID":"e3b6aa7b-cfb2-48b4-a762-e3dd5cbb1f41","Type":"ContainerStarted","Data":"86b4063c01e650fcdd44cd57441f70aa86201b55b2d446ca8fe833e4e23fffad"} Nov 01 20:11:16.232534 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:16.232405 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-7f0c86-84cd8cd7d9-pfqdn" Nov 01 20:11:16.234149 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:16.234124 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7f0c86-946796b95-bbvrc" event={"ID":"657cdb6c-3591-4340-8737-c15687f3f32b","Type":"ContainerStarted","Data":"f73febf61804860b2af76c50e3f160554df68b3c9df63dd029d855a19409ec18"} Nov 01 20:11:16.234265 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:16.234164 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7f0c86-946796b95-bbvrc" event={"ID":"657cdb6c-3591-4340-8737-c15687f3f32b","Type":"ContainerStarted","Data":"63c36b375587bd3aea8ca3113751b84561141d5b9fdaecb9357aa1ea6e8acc43"} Nov 01 20:11:16.234265 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:16.234206 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7f0c86-946796b95-bbvrc" Nov 01 20:11:16.249399 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:16.249352 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-7f0c86-84cd8cd7d9-pfqdn" podStartSLOduration=1.249341162 podStartE2EDuration="1.249341162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:11:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:11:16.248887247 +0000 UTC m=+722.022047759" watchObservedRunningTime="2026-11-01 20:11:16.249341162 +0000 UTC m=+722.022501673" Nov 01 20:11:16.264013 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:16.263964 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7f0c86-946796b95-bbvrc" podStartSLOduration=1.263952238 podStartE2EDuration="1.263952238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:11:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:11:16.26367831 +0000 UTC m=+722.036838822" watchObservedRunningTime="2026-11-01 20:11:16.263952238 +0000 UTC m=+722.037112783" Nov 01 20:11:27.244025 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:27.243950 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7f0c86-946796b95-bbvrc" Nov 01 20:11:27.244393 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:27.244196 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-7f0c86-84cd8cd7d9-pfqdn" Nov 01 20:11:35.843520 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:11:35.843488 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:11:44.539463 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:44.539416 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f" podUID="9e1d573b-e388-4d7b-b47a-8a01467c03b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:11:44.540397 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:44.540356 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-w4-0aa61b-kserve-6674df959d-2lm95" podUID="b7f35703-1219-4dd4-8bbd-631fb7303724" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:11:44.807364 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:44.806693 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/simulator-w4-0aa61b-kserve-6774594d4b-qmjz8" podUID="09414dc0-7742-48fa-a4b5-571199d99e15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:11:44.807552 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:44.807451 5935 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e27ae7/distinct-w2-e27ae7-kserve-7df8547d56-9qgdk" podUID="4a948974-537a-49da-85e7-60adf68d1655" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:11:45.203351 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:45.202642 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e27ae7/distinct-w2-e27ae7-kserve-7df8547d56-9qgdk" podUID="4a948974-537a-49da-85e7-60adf68d1655" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:11:45.275478 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:45.275430 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e27ae7/distinct-2-w2-e27ae7-kserve-6bf4d46475-jdjhp" podUID="b916d612-a355-4525-8143-9e3a47ead6d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:11:45.546485 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:45.541544 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f" podUID="9e1d573b-e388-4d7b-b47a-8a01467c03b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 01 20:11:45.546485 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:45.542725 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3a6480/distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f" Nov 01 20:11:45.546485 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:45.543534 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-w4-0aa61b-kserve-6674df959d-2lm95" podUID="b7f35703-1219-4dd4-8bbd-631fb7303724" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:11:45.546485 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:45.543858 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-w4-0aa61b-kserve-6674df959d-2lm95" Nov 01 20:11:45.840365 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:45.813020 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/simulator-w4-0aa61b-kserve-6774594d4b-qmjz8" podUID="09414dc0-7742-48fa-a4b5-571199d99e15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:11:45.840365 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:45.813405 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-0aa61b/simulator-w4-0aa61b-kserve-6774594d4b-qmjz8" Nov 01 20:11:46.049811 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:46.049780 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3a6480/distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f" Nov 01 20:11:46.150360 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:46.144122 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-w4-0aa61b-kserve-6674df959d-2lm95" Nov 01 20:11:46.150360 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:46.148903 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-0aa61b/simulator-w4-0aa61b-kserve-6774594d4b-qmjz8" Nov 01 20:11:54.053436 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:54.053387 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f" podUID="9e1d573b-e388-4d7b-b47a-8a01467c03b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 20:11:54.210311 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:54.206409 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e27ae7/distinct-w2-e27ae7-kserve-7df8547d56-9qgdk" podUID="4a948974-537a-49da-85e7-60adf68d1655" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 01 20:11:55.097308 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:55.073679 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f" podUID="9e1d573b-e388-4d7b-b47a-8a01467c03b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded" Nov 01 20:11:55.113428 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:55.113393 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3a6480/distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f" Nov 01 20:11:55.484109 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:55.483838 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3a6480/distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f" Nov 01 20:11:55.530601 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:55.530573 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-638f19/praxis-route-638f19-kserve-79cd6d56c-2gqpt"] Nov 01 20:11:55.650418 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:55.650373 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-638f19/praxis-route-638f19-kserve-79cd6d56c-2gqpt"] Nov 01 20:11:55.662309 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:55.656436 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-638f19/praxis-route-638f19-kserve-79cd6d56c-2gqpt" Nov 01 20:11:55.693349 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:55.691556 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2140ea23-2136-478a-b9d3-842a0898d794-tls-certs\") pod \"praxis-route-638f19-kserve-79cd6d56c-2gqpt\" (UID: \"2140ea23-2136-478a-b9d3-842a0898d794\") " pod="ai-tenant-e2e-praxis-a-638f19/praxis-route-638f19-kserve-79cd6d56c-2gqpt" Nov 01 20:11:55.737733 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:55.737665 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6sng\" (UniqueName: \"kubernetes.io/projected/2140ea23-2136-478a-b9d3-842a0898d794-kube-api-access-c6sng\") pod \"praxis-route-638f19-kserve-79cd6d56c-2gqpt\" (UID: \"2140ea23-2136-478a-b9d3-842a0898d794\") " pod="ai-tenant-e2e-praxis-a-638f19/praxis-route-638f19-kserve-79cd6d56c-2gqpt" Nov 01 20:11:55.738002 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:55.737980 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2140ea23-2136-478a-b9d3-842a0898d794-dshm\") pod \"praxis-route-638f19-kserve-79cd6d56c-2gqpt\" (UID: \"2140ea23-2136-478a-b9d3-842a0898d794\") " pod="ai-tenant-e2e-praxis-a-638f19/praxis-route-638f19-kserve-79cd6d56c-2gqpt" Nov 01 20:11:55.738147 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:55.738131 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2140ea23-2136-478a-b9d3-842a0898d794-tmp-dir\") pod \"praxis-route-638f19-kserve-79cd6d56c-2gqpt\" (UID: \"2140ea23-2136-478a-b9d3-842a0898d794\") " pod="ai-tenant-e2e-praxis-a-638f19/praxis-route-638f19-kserve-79cd6d56c-2gqpt" Nov 01 20:11:55.738325 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:55.738296 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2140ea23-2136-478a-b9d3-842a0898d794-model-cache\") pod \"praxis-route-638f19-kserve-79cd6d56c-2gqpt\" (UID: \"2140ea23-2136-478a-b9d3-842a0898d794\") " pod="ai-tenant-e2e-praxis-a-638f19/praxis-route-638f19-kserve-79cd6d56c-2gqpt" Nov 01 20:11:55.738477 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:55.738464 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2140ea23-2136-478a-b9d3-842a0898d794-home\") pod \"praxis-route-638f19-kserve-79cd6d56c-2gqpt\" (UID: \"2140ea23-2136-478a-b9d3-842a0898d794\") " pod="ai-tenant-e2e-praxis-a-638f19/praxis-route-638f19-kserve-79cd6d56c-2gqpt" Nov 01 20:11:55.761320 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:55.759362 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-638f19\"/\"default-dockercfg-4sgmc\"" Nov 01 20:11:55.761320 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:55.760073 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-638f19\"/\"kube-root-ca.crt\"" Nov 01 20:11:55.761320 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:55.760140 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-638f19\"/\"openshift-service-ca.crt\"" Nov 01 20:11:55.798976 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:55.798612 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-638f19\"/\"praxis-route-638f19-kserve-self-signed-certs\"" Nov 01 20:11:55.939004 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:55.938958 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2140ea23-2136-478a-b9d3-842a0898d794-tls-certs\") pod \"praxis-route-638f19-kserve-79cd6d56c-2gqpt\" (UID: \"2140ea23-2136-478a-b9d3-842a0898d794\") " pod="ai-tenant-e2e-praxis-a-638f19/praxis-route-638f19-kserve-79cd6d56c-2gqpt" Nov 01 20:11:55.944453 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:55.944419 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6sng\" (UniqueName: \"kubernetes.io/projected/2140ea23-2136-478a-b9d3-842a0898d794-kube-api-access-c6sng\") pod \"praxis-route-638f19-kserve-79cd6d56c-2gqpt\" (UID: \"2140ea23-2136-478a-b9d3-842a0898d794\") " pod="ai-tenant-e2e-praxis-a-638f19/praxis-route-638f19-kserve-79cd6d56c-2gqpt" Nov 01 20:11:55.944795 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:55.944766 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2140ea23-2136-478a-b9d3-842a0898d794-dshm\") pod \"praxis-route-638f19-kserve-79cd6d56c-2gqpt\" (UID: \"2140ea23-2136-478a-b9d3-842a0898d794\") " pod="ai-tenant-e2e-praxis-a-638f19/praxis-route-638f19-kserve-79cd6d56c-2gqpt" Nov 01 20:11:55.944969 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:55.944939 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2140ea23-2136-478a-b9d3-842a0898d794-tmp-dir\") pod \"praxis-route-638f19-kserve-79cd6d56c-2gqpt\" (UID: \"2140ea23-2136-478a-b9d3-842a0898d794\") " pod="ai-tenant-e2e-praxis-a-638f19/praxis-route-638f19-kserve-79cd6d56c-2gqpt" Nov 01 20:11:55.945332 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:55.945310 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2140ea23-2136-478a-b9d3-842a0898d794-model-cache\") pod \"praxis-route-638f19-kserve-79cd6d56c-2gqpt\" (UID: \"2140ea23-2136-478a-b9d3-842a0898d794\") " pod="ai-tenant-e2e-praxis-a-638f19/praxis-route-638f19-kserve-79cd6d56c-2gqpt" Nov 01 20:11:55.949458 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:55.947807 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2140ea23-2136-478a-b9d3-842a0898d794-tmp-dir\") pod \"praxis-route-638f19-kserve-79cd6d56c-2gqpt\" (UID: \"2140ea23-2136-478a-b9d3-842a0898d794\") " pod="ai-tenant-e2e-praxis-a-638f19/praxis-route-638f19-kserve-79cd6d56c-2gqpt" Nov 01 20:11:55.949458 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:55.948357 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2140ea23-2136-478a-b9d3-842a0898d794-model-cache\") pod \"praxis-route-638f19-kserve-79cd6d56c-2gqpt\" (UID: \"2140ea23-2136-478a-b9d3-842a0898d794\") " pod="ai-tenant-e2e-praxis-a-638f19/praxis-route-638f19-kserve-79cd6d56c-2gqpt" Nov 01 20:11:55.949917 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:55.949875 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2140ea23-2136-478a-b9d3-842a0898d794-home\") pod \"praxis-route-638f19-kserve-79cd6d56c-2gqpt\" (UID: \"2140ea23-2136-478a-b9d3-842a0898d794\") " pod="ai-tenant-e2e-praxis-a-638f19/praxis-route-638f19-kserve-79cd6d56c-2gqpt" Nov 01 20:11:55.950225 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:55.950199 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2140ea23-2136-478a-b9d3-842a0898d794-home\") pod \"praxis-route-638f19-kserve-79cd6d56c-2gqpt\" (UID: \"2140ea23-2136-478a-b9d3-842a0898d794\") " pod="ai-tenant-e2e-praxis-a-638f19/praxis-route-638f19-kserve-79cd6d56c-2gqpt" Nov 01 20:11:55.987779 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:55.969271 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2140ea23-2136-478a-b9d3-842a0898d794-dshm\") pod \"praxis-route-638f19-kserve-79cd6d56c-2gqpt\" (UID: \"2140ea23-2136-478a-b9d3-842a0898d794\") " pod="ai-tenant-e2e-praxis-a-638f19/praxis-route-638f19-kserve-79cd6d56c-2gqpt" Nov 01 20:11:55.987779 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:55.981196 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2140ea23-2136-478a-b9d3-842a0898d794-tls-certs\") pod \"praxis-route-638f19-kserve-79cd6d56c-2gqpt\" (UID: \"2140ea23-2136-478a-b9d3-842a0898d794\") " pod="ai-tenant-e2e-praxis-a-638f19/praxis-route-638f19-kserve-79cd6d56c-2gqpt" Nov 01 20:11:55.999079 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:55.999016 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6sng\" (UniqueName: \"kubernetes.io/projected/2140ea23-2136-478a-b9d3-842a0898d794-kube-api-access-c6sng\") pod \"praxis-route-638f19-kserve-79cd6d56c-2gqpt\" (UID: \"2140ea23-2136-478a-b9d3-842a0898d794\") " pod="ai-tenant-e2e-praxis-a-638f19/praxis-route-638f19-kserve-79cd6d56c-2gqpt" Nov 01 20:11:56.054794 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:56.054758 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-638f19/praxis-route-638f19-kserve-79cd6d56c-2gqpt" Nov 01 20:11:56.725706 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:56.724824 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-638f19/praxis-route-638f19-kserve-79cd6d56c-2gqpt"] Nov 01 20:11:56.730096 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:11:56.730021 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2140ea23_2136_478a_b9d3_842a0898d794.slice/crio-5d6fd998c9da5d971696796c4c5fe3bf07ed13dec572fad2947385972c43a864 WatchSource:0}: Error finding container 5d6fd998c9da5d971696796c4c5fe3bf07ed13dec572fad2947385972c43a864: Status 404 returned error can't find the container with id 5d6fd998c9da5d971696796c4c5fe3bf07ed13dec572fad2947385972c43a864 Nov 01 20:11:56.736432 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:56.736406 5935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-ljtcq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 20:11:56.736541 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:56.736455 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-ljtcq" podUID="03014616-4520-48bb-a548-66485577e26c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:11:56.931548 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:56.931513 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-638f19/praxis-route-638f19-kserve-79cd6d56c-2gqpt" event={"ID":"2140ea23-2136-478a-b9d3-842a0898d794","Type":"ContainerStarted","Data":"aa471da2c839266c20e8e61718d6b4791edc45e0101a1ba02ee7a53f37de4a3d"} Nov 01 20:11:56.931711 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:56.931559 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-638f19/praxis-route-638f19-kserve-79cd6d56c-2gqpt" event={"ID":"2140ea23-2136-478a-b9d3-842a0898d794","Type":"ContainerStarted","Data":"5d6fd998c9da5d971696796c4c5fe3bf07ed13dec572fad2947385972c43a864"} Nov 01 20:11:56.951101 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:56.951040 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-638f19/praxis-route-638f19-kserve-79cd6d56c-2gqpt" podStartSLOduration=1.951021447 podStartE2EDuration="1.951021447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:11:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:11:56.948180837 +0000 UTC m=+762.721341349" watchObservedRunningTime="2026-11-01 20:11:56.951021447 +0000 UTC m=+762.724181960" Nov 01 20:11:57.055589 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:11:57.055496 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-638f19/praxis-route-638f19-kserve-79cd6d56c-2gqpt" Nov 01 20:12:06.062297 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:06.062247 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-638f19/praxis-route-638f19-kserve-79cd6d56c-2gqpt" Nov 01 20:12:06.090789 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:12:06.090748 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:12:06.679637 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:06.679602 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-638f19/praxis-route-638f19-kserve-79cd6d56c-2gqpt" Nov 01 20:12:06.819427 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:06.819387 5935 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 20:12:06.820611 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:06.820573 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k" podUID="7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:12:07.188567 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:07.188528 5935 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 20:12:07.189251 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:07.189214 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s" podUID="e8bf3d0d-c668-44f1-9238-746e548faba0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:12:07.269252 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:07.269213 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-638f19/praxis-route-638f19-kserve-79cd6d56c-2gqpt" Nov 01 20:12:35.844494 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:12:35.844457 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:12:48.276658 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:48.276611 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-638f19/praxis-route-638f19-kserve-79cd6d56c-2gqpt" podUID="2140ea23-2136-478a-b9d3-842a0898d794" containerName="main" probeResult="failure" output="Get \"https://10.133.0.71:8000/ready\": context deadline exceeded" Nov 01 20:12:54.008736 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:54.008696 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxpr5s"] Nov 01 20:12:54.013247 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:54.013222 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxpr5s" Nov 01 20:12:54.020229 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:54.020202 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5370bb\"/\"openshift-service-ca.crt\"" Nov 01 20:12:54.020229 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:54.020204 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5370bb\"/\"routing-overlay\"" Nov 01 20:12:54.020449 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:54.020204 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5370bb\"/\"payload-processing-external-model-plugins-e2e-extmod-b-5370bb\"" Nov 01 20:12:54.020449 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:54.020245 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-5370bb\"/\"kube-root-ca.crt\"" Nov 01 20:12:54.024471 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:54.024432 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-5370bb\"/\"agc-cred-142f6c4e991f\"" Nov 01 20:12:54.024471 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:54.024446 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-5370bb\"/\"agc-cred-0d951a30aed7\"" Nov 01 20:12:54.024471 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:54.024465 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-5370bb\"/\"agc-cred-92ebf408b9f6\"" Nov 01 20:12:54.024666 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:54.024433 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-5370bb\"/\"agc-cred-cb46214ead48\"" Nov 01 20:12:54.024666 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:54.024565 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-5370bb\"/\"payload-processing-external-model-e2e--67d773b9-dockercfg-w8mwq\"" Nov 01 20:12:54.024767 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:54.024744 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxpr5s"] Nov 01 20:12:54.147851 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:54.147822 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1420bd3e-aa71-43bb-98a9-00dbfc4812b1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxpr5s\" (UID: \"1420bd3e-aa71-43bb-98a9-00dbfc4812b1\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxpr5s" Nov 01 20:12:54.147982 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:54.147957 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1420bd3e-aa71-43bb-98a9-00dbfc4812b1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxpr5s\" (UID: \"1420bd3e-aa71-43bb-98a9-00dbfc4812b1\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxpr5s" Nov 01 20:12:54.148053 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:54.148036 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htwx4\" (UniqueName: \"kubernetes.io/projected/1420bd3e-aa71-43bb-98a9-00dbfc4812b1-kube-api-access-htwx4\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxpr5s\" (UID: \"1420bd3e-aa71-43bb-98a9-00dbfc4812b1\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxpr5s" Nov 01 20:12:54.148103 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:54.148093 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1420bd3e-aa71-43bb-98a9-00dbfc4812b1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxpr5s\" (UID: \"1420bd3e-aa71-43bb-98a9-00dbfc4812b1\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxpr5s" Nov 01 20:12:54.248667 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:54.248639 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1420bd3e-aa71-43bb-98a9-00dbfc4812b1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxpr5s\" (UID: \"1420bd3e-aa71-43bb-98a9-00dbfc4812b1\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxpr5s" Nov 01 20:12:54.248787 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:54.248710 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htwx4\" (UniqueName: \"kubernetes.io/projected/1420bd3e-aa71-43bb-98a9-00dbfc4812b1-kube-api-access-htwx4\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxpr5s\" (UID: \"1420bd3e-aa71-43bb-98a9-00dbfc4812b1\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxpr5s" Nov 01 20:12:54.248787 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:54.248767 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1420bd3e-aa71-43bb-98a9-00dbfc4812b1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxpr5s\" (UID: \"1420bd3e-aa71-43bb-98a9-00dbfc4812b1\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxpr5s" Nov 01 20:12:54.248881 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:54.248837 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1420bd3e-aa71-43bb-98a9-00dbfc4812b1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxpr5s\" (UID: \"1420bd3e-aa71-43bb-98a9-00dbfc4812b1\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxpr5s" Nov 01 20:12:54.249274 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:54.249240 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1420bd3e-aa71-43bb-98a9-00dbfc4812b1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxpr5s\" (UID: \"1420bd3e-aa71-43bb-98a9-00dbfc4812b1\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxpr5s" Nov 01 20:12:54.249512 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:54.249483 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1420bd3e-aa71-43bb-98a9-00dbfc4812b1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxpr5s\" (UID: \"1420bd3e-aa71-43bb-98a9-00dbfc4812b1\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxpr5s" Nov 01 20:12:54.251439 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:54.251420 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1420bd3e-aa71-43bb-98a9-00dbfc4812b1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxpr5s\" (UID: \"1420bd3e-aa71-43bb-98a9-00dbfc4812b1\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxpr5s" Nov 01 20:12:54.256308 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:54.256273 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htwx4\" (UniqueName: \"kubernetes.io/projected/1420bd3e-aa71-43bb-98a9-00dbfc4812b1-kube-api-access-htwx4\") pod \"payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxpr5s\" (UID: \"1420bd3e-aa71-43bb-98a9-00dbfc4812b1\") " pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxpr5s" Nov 01 20:12:54.343202 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:54.343145 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxpr5s" Nov 01 20:12:54.640008 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:54.639968 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxpr5s"] Nov 01 20:12:54.743055 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:54.743028 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-29b57c-7b87bbfff-2nmvw"] Nov 01 20:12:54.747565 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:54.747540 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-29b57c-7b87bbfff-2nmvw" Nov 01 20:12:54.761439 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:54.760876 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-29b57c\"" Nov 01 20:12:54.761439 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:54.761026 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-29b57c-dockercfg-7dpnq\"" Nov 01 20:12:54.762637 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:54.762613 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-29b57c-7b87bbfff-2nmvw"] Nov 01 20:12:54.855054 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:54.854964 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9k2l\" (UniqueName: \"kubernetes.io/projected/6465f212-e931-4b70-8b1a-a1ec15850515-kube-api-access-w9k2l\") pod \"payload-pre-processing-e2e-extmod-a-29b57c-7b87bbfff-2nmvw\" (UID: \"6465f212-e931-4b70-8b1a-a1ec15850515\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-29b57c-7b87bbfff-2nmvw" Nov 01 20:12:54.855281 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:54.855158 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6465f212-e931-4b70-8b1a-a1ec15850515-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-29b57c-7b87bbfff-2nmvw\" (UID: \"6465f212-e931-4b70-8b1a-a1ec15850515\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-29b57c-7b87bbfff-2nmvw" Nov 01 20:12:54.956483 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:54.956445 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9k2l\" (UniqueName: \"kubernetes.io/projected/6465f212-e931-4b70-8b1a-a1ec15850515-kube-api-access-w9k2l\") pod \"payload-pre-processing-e2e-extmod-a-29b57c-7b87bbfff-2nmvw\" (UID: \"6465f212-e931-4b70-8b1a-a1ec15850515\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-29b57c-7b87bbfff-2nmvw" Nov 01 20:12:54.956664 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:54.956580 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6465f212-e931-4b70-8b1a-a1ec15850515-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-29b57c-7b87bbfff-2nmvw\" (UID: \"6465f212-e931-4b70-8b1a-a1ec15850515\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-29b57c-7b87bbfff-2nmvw" Nov 01 20:12:54.957580 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:54.957550 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6465f212-e931-4b70-8b1a-a1ec15850515-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-29b57c-7b87bbfff-2nmvw\" (UID: \"6465f212-e931-4b70-8b1a-a1ec15850515\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-29b57c-7b87bbfff-2nmvw" Nov 01 20:12:54.965056 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:54.965032 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9k2l\" (UniqueName: \"kubernetes.io/projected/6465f212-e931-4b70-8b1a-a1ec15850515-kube-api-access-w9k2l\") pod \"payload-pre-processing-e2e-extmod-a-29b57c-7b87bbfff-2nmvw\" (UID: \"6465f212-e931-4b70-8b1a-a1ec15850515\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-29b57c-7b87bbfff-2nmvw" Nov 01 20:12:55.066509 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:55.066475 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-29b57c-7b87bbfff-2nmvw" Nov 01 20:12:55.353561 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:55.353537 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-29b57c-7b87bbfff-2nmvw"] Nov 01 20:12:55.355341 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:12:55.355312 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6465f212_e931_4b70_8b1a_a1ec15850515.slice/crio-acfbb3063db32daddb0e0024d27d3b0bf0b2437573349b8c0ce1683922194841 WatchSource:0}: Error finding container acfbb3063db32daddb0e0024d27d3b0bf0b2437573349b8c0ce1683922194841: Status 404 returned error can't find the container with id acfbb3063db32daddb0e0024d27d3b0bf0b2437573349b8c0ce1683922194841 Nov 01 20:12:55.561562 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:55.561528 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-29b57c-7b87bbfff-2nmvw" event={"ID":"6465f212-e931-4b70-8b1a-a1ec15850515","Type":"ContainerStarted","Data":"01559ca3befe74277b2e30d17c2c8ecde908e6f86904ad7e3e0102bfa75abd94"} Nov 01 20:12:55.561751 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:55.561593 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-29b57c-7b87bbfff-2nmvw" Nov 01 20:12:55.561751 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:55.561621 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-29b57c-7b87bbfff-2nmvw" event={"ID":"6465f212-e931-4b70-8b1a-a1ec15850515","Type":"ContainerStarted","Data":"acfbb3063db32daddb0e0024d27d3b0bf0b2437573349b8c0ce1683922194841"} Nov 01 20:12:55.564025 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:55.563991 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxpr5s" event={"ID":"1420bd3e-aa71-43bb-98a9-00dbfc4812b1","Type":"ContainerStarted","Data":"c47c77e5362407555e3a9c9592b6227f4e3b9eaace029ace5458aed73c7ee9eb"} Nov 01 20:12:55.564150 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:55.564033 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxpr5s" event={"ID":"1420bd3e-aa71-43bb-98a9-00dbfc4812b1","Type":"ContainerStarted","Data":"a39c303d61fff772ec0ff89597e181d641deb23f9abe2934a0272329b5723897"} Nov 01 20:12:55.564530 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:55.564515 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxpr5s" Nov 01 20:12:55.577932 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:55.577890 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-29b57c-7b87bbfff-2nmvw" podStartSLOduration=1.5778784209999999 podStartE2EDuration="1.577878421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:12:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:12:55.576693534 +0000 UTC m=+821.349854042" watchObservedRunningTime="2026-11-01 20:12:55.577878421 +0000 UTC m=+821.351038933" Nov 01 20:12:55.593297 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:55.593232 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxpr5s" podStartSLOduration=2.593218781 podStartE2EDuration="2.593218781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:12:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:12:55.591493708 +0000 UTC m=+821.364654225" watchObservedRunningTime="2026-11-01 20:12:55.593218781 +0000 UTC m=+821.366379302" Nov 01 20:12:58.701608 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:58.701577 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-29b57c-69b9ffd5f7-rj47z"] Nov 01 20:12:58.706097 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:58.706071 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-29b57c-69b9ffd5f7-rj47z" Nov 01 20:12:58.715146 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:58.715011 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-29b57c-serving-cert\"" Nov 01 20:12:58.723656 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:58.719900 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-29b57c-69b9ffd5f7-rj47z"] Nov 01 20:12:58.801162 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:58.801137 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42fe7bf5-319c-449a-aa5a-28b1e2d1f008-maas-api-tls\") pod \"maas-api-e2e-extmod-a-29b57c-69b9ffd5f7-rj47z\" (UID: \"42fe7bf5-319c-449a-aa5a-28b1e2d1f008\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-29b57c-69b9ffd5f7-rj47z" Nov 01 20:12:58.801332 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:58.801195 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzl4t\" (UniqueName: \"kubernetes.io/projected/42fe7bf5-319c-449a-aa5a-28b1e2d1f008-kube-api-access-dzl4t\") pod \"maas-api-e2e-extmod-a-29b57c-69b9ffd5f7-rj47z\" (UID: \"42fe7bf5-319c-449a-aa5a-28b1e2d1f008\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-29b57c-69b9ffd5f7-rj47z" Nov 01 20:12:58.801402 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:58.801346 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42fe7bf5-319c-449a-aa5a-28b1e2d1f008-metrics-certs\") pod \"maas-api-e2e-extmod-a-29b57c-69b9ffd5f7-rj47z\" (UID: \"42fe7bf5-319c-449a-aa5a-28b1e2d1f008\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-29b57c-69b9ffd5f7-rj47z" Nov 01 20:12:58.801402 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:58.801382 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42fe7bf5-319c-449a-aa5a-28b1e2d1f008-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-29b57c-69b9ffd5f7-rj47z\" (UID: \"42fe7bf5-319c-449a-aa5a-28b1e2d1f008\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-29b57c-69b9ffd5f7-rj47z" Nov 01 20:12:58.902625 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:58.902588 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzl4t\" (UniqueName: \"kubernetes.io/projected/42fe7bf5-319c-449a-aa5a-28b1e2d1f008-kube-api-access-dzl4t\") pod \"maas-api-e2e-extmod-a-29b57c-69b9ffd5f7-rj47z\" (UID: \"42fe7bf5-319c-449a-aa5a-28b1e2d1f008\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-29b57c-69b9ffd5f7-rj47z" Nov 01 20:12:58.902827 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:58.902715 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42fe7bf5-319c-449a-aa5a-28b1e2d1f008-metrics-certs\") pod \"maas-api-e2e-extmod-a-29b57c-69b9ffd5f7-rj47z\" (UID: \"42fe7bf5-319c-449a-aa5a-28b1e2d1f008\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-29b57c-69b9ffd5f7-rj47z" Nov 01 20:12:58.902827 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:58.902747 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42fe7bf5-319c-449a-aa5a-28b1e2d1f008-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-29b57c-69b9ffd5f7-rj47z\" (UID: \"42fe7bf5-319c-449a-aa5a-28b1e2d1f008\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-29b57c-69b9ffd5f7-rj47z" Nov 01 20:12:58.902955 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:58.902844 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42fe7bf5-319c-449a-aa5a-28b1e2d1f008-maas-api-tls\") pod \"maas-api-e2e-extmod-a-29b57c-69b9ffd5f7-rj47z\" (UID: \"42fe7bf5-319c-449a-aa5a-28b1e2d1f008\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-29b57c-69b9ffd5f7-rj47z" Nov 01 20:12:58.903633 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:58.903600 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42fe7bf5-319c-449a-aa5a-28b1e2d1f008-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-29b57c-69b9ffd5f7-rj47z\" (UID: \"42fe7bf5-319c-449a-aa5a-28b1e2d1f008\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-29b57c-69b9ffd5f7-rj47z" Nov 01 20:12:58.908705 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:58.908681 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42fe7bf5-319c-449a-aa5a-28b1e2d1f008-maas-api-tls\") pod \"maas-api-e2e-extmod-a-29b57c-69b9ffd5f7-rj47z\" (UID: \"42fe7bf5-319c-449a-aa5a-28b1e2d1f008\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-29b57c-69b9ffd5f7-rj47z" Nov 01 20:12:58.908812 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:58.908710 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42fe7bf5-319c-449a-aa5a-28b1e2d1f008-metrics-certs\") pod \"maas-api-e2e-extmod-a-29b57c-69b9ffd5f7-rj47z\" (UID: \"42fe7bf5-319c-449a-aa5a-28b1e2d1f008\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-29b57c-69b9ffd5f7-rj47z" Nov 01 20:12:58.912849 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:58.912826 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzl4t\" (UniqueName: \"kubernetes.io/projected/42fe7bf5-319c-449a-aa5a-28b1e2d1f008-kube-api-access-dzl4t\") pod \"maas-api-e2e-extmod-a-29b57c-69b9ffd5f7-rj47z\" (UID: \"42fe7bf5-319c-449a-aa5a-28b1e2d1f008\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-29b57c-69b9ffd5f7-rj47z" Nov 01 20:12:59.028013 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:59.027943 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-29b57c-69b9ffd5f7-rj47z" Nov 01 20:12:59.329992 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:59.329959 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-29b57c-69b9ffd5f7-rj47z"] Nov 01 20:12:59.334485 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:59.334463 5935 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 20:12:59.593941 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:12:59.593859 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-29b57c-69b9ffd5f7-rj47z" event={"ID":"42fe7bf5-319c-449a-aa5a-28b1e2d1f008","Type":"ContainerStarted","Data":"56b668f62ca601b84fa2495308f4e5fd0a63b89f233515ab6786bc560c59c067"} Nov 01 20:13:00.601580 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:00.601546 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-29b57c-69b9ffd5f7-rj47z" event={"ID":"42fe7bf5-319c-449a-aa5a-28b1e2d1f008","Type":"ContainerStarted","Data":"08f680832649f004297c84a621796145c4a15a47c9f33341732d2623b209fcdd"} Nov 01 20:13:00.601979 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:00.601617 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-29b57c-69b9ffd5f7-rj47z" Nov 01 20:13:00.618582 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:00.618534 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-29b57c-69b9ffd5f7-rj47z" podStartSLOduration=2.120270138 podStartE2EDuration="2.618519687s" totalImagesPullingTime="498.249549ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:12:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:13:00.617007159 +0000 UTC m=+826.390167686" watchObservedRunningTime="2026-11-01 20:13:00.618519687 +0000 UTC m=+826.391680203" Nov 01 20:13:04.670966 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:04.670927 5935 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 20:13:04.671585 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:04.671547 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb" podUID="f8249130-8b25-468a-9904-2564d38fdb15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:13:05.844227 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:13:05.844193 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:13:06.575467 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:06.575440 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-29b57c-7b87bbfff-2nmvw" Nov 01 20:13:06.614894 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:06.614865 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-29b57c-69b9ffd5f7-rj47z" Nov 01 20:13:07.583446 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:07.583416 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxpr5s" Nov 01 20:13:12.665948 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:12.665917 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-638f19/praxis-route-638f19-kserve-79cd6d56c-2gqpt"] Nov 01 20:13:14.496368 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:14.496337 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-bccce1-5b9f8747f9-rvbs2"] Nov 01 20:13:14.500809 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:14.500785 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-bccce1-5b9f8747f9-rvbs2" Nov 01 20:13:14.506388 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:14.506366 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-bccce1-5b9f8747f9-rvbs2"] Nov 01 20:13:14.561227 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:14.561202 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-bccce1\"" Nov 01 20:13:14.562003 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:14.561986 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-bccce1-dockercfg-v7d5z\"" Nov 01 20:13:14.598255 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:14.598219 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de43c730-bc16-43b6-8cbb-c32a14622bb7-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-bccce1-5b9f8747f9-rvbs2\" (UID: \"de43c730-bc16-43b6-8cbb-c32a14622bb7\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-bccce1-5b9f8747f9-rvbs2" Nov 01 20:13:14.598463 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:14.598407 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trmkv\" (UniqueName: \"kubernetes.io/projected/de43c730-bc16-43b6-8cbb-c32a14622bb7-kube-api-access-trmkv\") pod \"payload-processing-e2e-extmod-b-bccce1-5b9f8747f9-rvbs2\" (UID: \"de43c730-bc16-43b6-8cbb-c32a14622bb7\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-bccce1-5b9f8747f9-rvbs2" Nov 01 20:13:14.700222 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:14.700186 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de43c730-bc16-43b6-8cbb-c32a14622bb7-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-bccce1-5b9f8747f9-rvbs2\" (UID: \"de43c730-bc16-43b6-8cbb-c32a14622bb7\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-bccce1-5b9f8747f9-rvbs2" Nov 01 20:13:14.700630 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:14.700602 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trmkv\" (UniqueName: \"kubernetes.io/projected/de43c730-bc16-43b6-8cbb-c32a14622bb7-kube-api-access-trmkv\") pod \"payload-processing-e2e-extmod-b-bccce1-5b9f8747f9-rvbs2\" (UID: \"de43c730-bc16-43b6-8cbb-c32a14622bb7\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-bccce1-5b9f8747f9-rvbs2" Nov 01 20:13:14.701454 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:14.701428 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de43c730-bc16-43b6-8cbb-c32a14622bb7-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-bccce1-5b9f8747f9-rvbs2\" (UID: \"de43c730-bc16-43b6-8cbb-c32a14622bb7\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-bccce1-5b9f8747f9-rvbs2" Nov 01 20:13:14.710507 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:14.710475 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trmkv\" (UniqueName: \"kubernetes.io/projected/de43c730-bc16-43b6-8cbb-c32a14622bb7-kube-api-access-trmkv\") pod \"payload-processing-e2e-extmod-b-bccce1-5b9f8747f9-rvbs2\" (UID: \"de43c730-bc16-43b6-8cbb-c32a14622bb7\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-bccce1-5b9f8747f9-rvbs2" Nov 01 20:13:14.819132 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:14.819042 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-bccce1-5b9f8747f9-rvbs2" Nov 01 20:13:15.111837 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:15.111808 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-bccce1-5b9f8747f9-rvbs2"] Nov 01 20:13:15.286419 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:15.286389 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0ebb0cff-7bcc55b8cf-89s86"] Nov 01 20:13:15.290250 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:15.290231 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0ebb0cff-7bcc55b8cf-89s86" Nov 01 20:13:15.299432 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:15.299408 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0ebb0cff-7bcc55b8cf-89s86"] Nov 01 20:13:15.358715 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:15.358677 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-0ebb0cff\"" Nov 01 20:13:15.358715 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:15.358704 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-0ebb0cff-dockercfg-lf8fx\"" Nov 01 20:13:15.414615 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:15.414518 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bda5c2cd-0ec3-4b3f-ad58-a51a91c81245-plugins-config-volume\") pod \"payload-processing-e2e-ait-0ebb0cff-7bcc55b8cf-89s86\" (UID: \"bda5c2cd-0ec3-4b3f-ad58-a51a91c81245\") " pod="openshift-ingress/payload-processing-e2e-ait-0ebb0cff-7bcc55b8cf-89s86" Nov 01 20:13:15.414773 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:15.414719 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njdf8\" (UniqueName: \"kubernetes.io/projected/bda5c2cd-0ec3-4b3f-ad58-a51a91c81245-kube-api-access-njdf8\") pod \"payload-processing-e2e-ait-0ebb0cff-7bcc55b8cf-89s86\" (UID: \"bda5c2cd-0ec3-4b3f-ad58-a51a91c81245\") " pod="openshift-ingress/payload-processing-e2e-ait-0ebb0cff-7bcc55b8cf-89s86" Nov 01 20:13:15.516396 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:15.516359 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njdf8\" (UniqueName: \"kubernetes.io/projected/bda5c2cd-0ec3-4b3f-ad58-a51a91c81245-kube-api-access-njdf8\") pod \"payload-processing-e2e-ait-0ebb0cff-7bcc55b8cf-89s86\" (UID: \"bda5c2cd-0ec3-4b3f-ad58-a51a91c81245\") " pod="openshift-ingress/payload-processing-e2e-ait-0ebb0cff-7bcc55b8cf-89s86" Nov 01 20:13:15.516797 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:15.516459 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bda5c2cd-0ec3-4b3f-ad58-a51a91c81245-plugins-config-volume\") pod \"payload-processing-e2e-ait-0ebb0cff-7bcc55b8cf-89s86\" (UID: \"bda5c2cd-0ec3-4b3f-ad58-a51a91c81245\") " pod="openshift-ingress/payload-processing-e2e-ait-0ebb0cff-7bcc55b8cf-89s86" Nov 01 20:13:15.517247 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:15.517228 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bda5c2cd-0ec3-4b3f-ad58-a51a91c81245-plugins-config-volume\") pod \"payload-processing-e2e-ait-0ebb0cff-7bcc55b8cf-89s86\" (UID: \"bda5c2cd-0ec3-4b3f-ad58-a51a91c81245\") " pod="openshift-ingress/payload-processing-e2e-ait-0ebb0cff-7bcc55b8cf-89s86" Nov 01 20:13:15.532961 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:15.532935 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njdf8\" (UniqueName: \"kubernetes.io/projected/bda5c2cd-0ec3-4b3f-ad58-a51a91c81245-kube-api-access-njdf8\") pod \"payload-processing-e2e-ait-0ebb0cff-7bcc55b8cf-89s86\" (UID: \"bda5c2cd-0ec3-4b3f-ad58-a51a91c81245\") " pod="openshift-ingress/payload-processing-e2e-ait-0ebb0cff-7bcc55b8cf-89s86" Nov 01 20:13:15.608898 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:15.608860 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0ebb0cff-7bcc55b8cf-89s86" Nov 01 20:13:15.746479 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:15.746444 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-bccce1-5b9f8747f9-rvbs2" event={"ID":"de43c730-bc16-43b6-8cbb-c32a14622bb7","Type":"ContainerStarted","Data":"41a73ca59d45b4e8ff308e407275a2ca069412843d1c74cb0b587e30976c0947"} Nov 01 20:13:15.746634 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:15.746503 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-bccce1-5b9f8747f9-rvbs2" event={"ID":"de43c730-bc16-43b6-8cbb-c32a14622bb7","Type":"ContainerStarted","Data":"73faa839da9433ee6e251be9cf345bf90534fab37e76cf24eccbbb3c6c369d49"} Nov 01 20:13:15.746634 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:15.746571 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-bccce1-5b9f8747f9-rvbs2" Nov 01 20:13:15.766465 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:15.766411 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-bccce1-5b9f8747f9-rvbs2" podStartSLOduration=1.7663941539999999 podStartE2EDuration="1.766394154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:13:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:13:15.764185627 +0000 UTC m=+841.537346138" watchObservedRunningTime="2026-11-01 20:13:15.766394154 +0000 UTC m=+841.539554667" Nov 01 20:13:15.887500 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:15.887454 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0ebb0cff-7bcc55b8cf-89s86"] Nov 01 20:13:15.888242 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:13:15.888212 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbda5c2cd_0ec3_4b3f_ad58_a51a91c81245.slice/crio-8a102e071f355ae410e9e1508a09470e46acd3c610240b20f3540350d0ccda56 WatchSource:0}: Error finding container 8a102e071f355ae410e9e1508a09470e46acd3c610240b20f3540350d0ccda56: Status 404 returned error can't find the container with id 8a102e071f355ae410e9e1508a09470e46acd3c610240b20f3540350d0ccda56 Nov 01 20:13:16.754204 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:16.754164 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0ebb0cff-7bcc55b8cf-89s86" event={"ID":"bda5c2cd-0ec3-4b3f-ad58-a51a91c81245","Type":"ContainerStarted","Data":"3e81b93475fcd03377eda8706dd1df7496e6a022e61d436aebb78b377bb7efbe"} Nov 01 20:13:16.754674 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:16.754380 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-0ebb0cff-7bcc55b8cf-89s86" Nov 01 20:13:16.754674 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:16.754429 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0ebb0cff-7bcc55b8cf-89s86" event={"ID":"bda5c2cd-0ec3-4b3f-ad58-a51a91c81245","Type":"ContainerStarted","Data":"8a102e071f355ae410e9e1508a09470e46acd3c610240b20f3540350d0ccda56"} Nov 01 20:13:16.772977 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:16.772932 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-0ebb0cff-7bcc55b8cf-89s86" podStartSLOduration=1.772919556 podStartE2EDuration="1.772919556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:13:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:13:16.769674697 +0000 UTC m=+842.542835213" watchObservedRunningTime="2026-11-01 20:13:16.772919556 +0000 UTC m=+842.546080080" Nov 01 20:13:21.746776 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:21.746726 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0ebb0cff-6d8669c646-8z9dg"] Nov 01 20:13:21.752156 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:21.752132 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0ebb0cff-6d8669c646-8z9dg" Nov 01 20:13:21.760626 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:21.760597 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0ebb0cff-6d8669c646-8z9dg"] Nov 01 20:13:21.766309 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:21.763129 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-0ebb0cff-serving-cert\"" Nov 01 20:13:21.892214 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:21.892181 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cxhb\" (UniqueName: \"kubernetes.io/projected/11752a2a-c301-4a45-89d8-470fcc7332f4-kube-api-access-4cxhb\") pod \"maas-api-e2e-ait-0ebb0cff-6d8669c646-8z9dg\" (UID: \"11752a2a-c301-4a45-89d8-470fcc7332f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0ebb0cff-6d8669c646-8z9dg" Nov 01 20:13:21.892214 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:21.892218 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11752a2a-c301-4a45-89d8-470fcc7332f4-metrics-certs\") pod \"maas-api-e2e-ait-0ebb0cff-6d8669c646-8z9dg\" (UID: \"11752a2a-c301-4a45-89d8-470fcc7332f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0ebb0cff-6d8669c646-8z9dg" Nov 01 20:13:21.892501 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:21.892405 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11752a2a-c301-4a45-89d8-470fcc7332f4-maas-api-tls\") pod \"maas-api-e2e-ait-0ebb0cff-6d8669c646-8z9dg\" (UID: \"11752a2a-c301-4a45-89d8-470fcc7332f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0ebb0cff-6d8669c646-8z9dg" Nov 01 20:13:21.892501 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:21.892447 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11752a2a-c301-4a45-89d8-470fcc7332f4-openshift-service-ca\") pod \"maas-api-e2e-ait-0ebb0cff-6d8669c646-8z9dg\" (UID: \"11752a2a-c301-4a45-89d8-470fcc7332f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0ebb0cff-6d8669c646-8z9dg" Nov 01 20:13:21.993618 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:21.993592 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11752a2a-c301-4a45-89d8-470fcc7332f4-openshift-service-ca\") pod \"maas-api-e2e-ait-0ebb0cff-6d8669c646-8z9dg\" (UID: \"11752a2a-c301-4a45-89d8-470fcc7332f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0ebb0cff-6d8669c646-8z9dg" Nov 01 20:13:21.993748 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:21.993675 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cxhb\" (UniqueName: \"kubernetes.io/projected/11752a2a-c301-4a45-89d8-470fcc7332f4-kube-api-access-4cxhb\") pod \"maas-api-e2e-ait-0ebb0cff-6d8669c646-8z9dg\" (UID: \"11752a2a-c301-4a45-89d8-470fcc7332f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0ebb0cff-6d8669c646-8z9dg" Nov 01 20:13:21.993748 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:21.993698 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11752a2a-c301-4a45-89d8-470fcc7332f4-metrics-certs\") pod \"maas-api-e2e-ait-0ebb0cff-6d8669c646-8z9dg\" (UID: \"11752a2a-c301-4a45-89d8-470fcc7332f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0ebb0cff-6d8669c646-8z9dg" Nov 01 20:13:21.993850 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:21.993806 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11752a2a-c301-4a45-89d8-470fcc7332f4-maas-api-tls\") pod \"maas-api-e2e-ait-0ebb0cff-6d8669c646-8z9dg\" (UID: \"11752a2a-c301-4a45-89d8-470fcc7332f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0ebb0cff-6d8669c646-8z9dg" Nov 01 20:13:21.994262 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:21.994240 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11752a2a-c301-4a45-89d8-470fcc7332f4-openshift-service-ca\") pod \"maas-api-e2e-ait-0ebb0cff-6d8669c646-8z9dg\" (UID: \"11752a2a-c301-4a45-89d8-470fcc7332f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0ebb0cff-6d8669c646-8z9dg" Nov 01 20:13:21.996154 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:21.996135 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11752a2a-c301-4a45-89d8-470fcc7332f4-maas-api-tls\") pod \"maas-api-e2e-ait-0ebb0cff-6d8669c646-8z9dg\" (UID: \"11752a2a-c301-4a45-89d8-470fcc7332f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0ebb0cff-6d8669c646-8z9dg" Nov 01 20:13:21.996246 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:21.996152 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11752a2a-c301-4a45-89d8-470fcc7332f4-metrics-certs\") pod \"maas-api-e2e-ait-0ebb0cff-6d8669c646-8z9dg\" (UID: \"11752a2a-c301-4a45-89d8-470fcc7332f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0ebb0cff-6d8669c646-8z9dg" Nov 01 20:13:22.001397 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:22.001343 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cxhb\" (UniqueName: \"kubernetes.io/projected/11752a2a-c301-4a45-89d8-470fcc7332f4-kube-api-access-4cxhb\") pod \"maas-api-e2e-ait-0ebb0cff-6d8669c646-8z9dg\" (UID: \"11752a2a-c301-4a45-89d8-470fcc7332f4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0ebb0cff-6d8669c646-8z9dg" Nov 01 20:13:22.073617 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:22.073582 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0ebb0cff-6d8669c646-8z9dg" Nov 01 20:13:22.364197 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:22.364163 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0ebb0cff-6d8669c646-8z9dg"] Nov 01 20:13:22.818076 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:22.818043 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0ebb0cff-6d8669c646-8z9dg" event={"ID":"11752a2a-c301-4a45-89d8-470fcc7332f4","Type":"ContainerStarted","Data":"a0767518eed69822e1e9fa7f4931a7c11ded079353b2324fddb51d1998acd913"} Nov 01 20:13:23.825967 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:23.825932 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0ebb0cff-6d8669c646-8z9dg" event={"ID":"11752a2a-c301-4a45-89d8-470fcc7332f4","Type":"ContainerStarted","Data":"e0224df5dab88cccf2a3259bbd0d3c9cf89c8c91b3fe9d5c61f5820eed3ffc09"} Nov 01 20:13:23.826515 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:23.826000 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0ebb0cff-6d8669c646-8z9dg" Nov 01 20:13:23.843127 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:23.843073 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0ebb0cff-6d8669c646-8z9dg" podStartSLOduration=2.410605356 podStartE2EDuration="2.843060262s" totalImagesPullingTime="432.454906ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:13:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:13:23.841115384 +0000 UTC m=+849.614275898" watchObservedRunningTime="2026-11-01 20:13:23.843060262 +0000 UTC m=+849.616220771" Nov 01 20:13:26.757766 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:26.757737 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-bccce1-5b9f8747f9-rvbs2" Nov 01 20:13:27.687148 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:27.687110 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-638f19/praxis-route-638f19-kserve-79cd6d56c-2gqpt" podUID="2140ea23-2136-478a-b9d3-842a0898d794" containerName="main" containerID="cri-o://aa471da2c839266c20e8e61718d6b4791edc45e0101a1ba02ee7a53f37de4a3d" gracePeriod=45 Nov 01 20:13:27.764444 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:27.764414 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-0ebb0cff-7bcc55b8cf-89s86" Nov 01 20:13:27.853425 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:27.853395 5935 generic.go:356] "Generic (PLEG): container finished" podID="2140ea23-2136-478a-b9d3-842a0898d794" containerID="aa471da2c839266c20e8e61718d6b4791edc45e0101a1ba02ee7a53f37de4a3d" exitCode=0 Nov 01 20:13:27.853582 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:27.853466 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-638f19/praxis-route-638f19-kserve-79cd6d56c-2gqpt" event={"ID":"2140ea23-2136-478a-b9d3-842a0898d794","Type":"ContainerDied","Data":"aa471da2c839266c20e8e61718d6b4791edc45e0101a1ba02ee7a53f37de4a3d"} Nov 01 20:13:27.963654 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:27.963627 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-638f19/praxis-route-638f19-kserve-79cd6d56c-2gqpt" Nov 01 20:13:28.068673 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:28.068642 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2140ea23-2136-478a-b9d3-842a0898d794-tls-certs\" (UniqueName: \"kubernetes.io/secret/2140ea23-2136-478a-b9d3-842a0898d794-tls-certs\") pod \"2140ea23-2136-478a-b9d3-842a0898d794\" (UID: \"2140ea23-2136-478a-b9d3-842a0898d794\") " Nov 01 20:13:28.068860 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:28.068731 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2140ea23-2136-478a-b9d3-842a0898d794-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2140ea23-2136-478a-b9d3-842a0898d794-model-cache\") pod \"2140ea23-2136-478a-b9d3-842a0898d794\" (UID: \"2140ea23-2136-478a-b9d3-842a0898d794\") " Nov 01 20:13:28.068860 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:28.068815 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2140ea23-2136-478a-b9d3-842a0898d794-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2140ea23-2136-478a-b9d3-842a0898d794-tmp-dir\") pod \"2140ea23-2136-478a-b9d3-842a0898d794\" (UID: \"2140ea23-2136-478a-b9d3-842a0898d794\") " Nov 01 20:13:28.068998 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:28.068906 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2140ea23-2136-478a-b9d3-842a0898d794-home\" (UniqueName: \"kubernetes.io/empty-dir/2140ea23-2136-478a-b9d3-842a0898d794-home\") pod \"2140ea23-2136-478a-b9d3-842a0898d794\" (UID: \"2140ea23-2136-478a-b9d3-842a0898d794\") " Nov 01 20:13:28.069055 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:28.068997 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2140ea23-2136-478a-b9d3-842a0898d794-kube-api-access-c6sng\" (UniqueName: \"kubernetes.io/projected/2140ea23-2136-478a-b9d3-842a0898d794-kube-api-access-c6sng\") pod \"2140ea23-2136-478a-b9d3-842a0898d794\" (UID: \"2140ea23-2136-478a-b9d3-842a0898d794\") " Nov 01 20:13:28.069055 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:28.069025 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2140ea23-2136-478a-b9d3-842a0898d794-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2140ea23-2136-478a-b9d3-842a0898d794-dshm\") pod \"2140ea23-2136-478a-b9d3-842a0898d794\" (UID: \"2140ea23-2136-478a-b9d3-842a0898d794\") " Nov 01 20:13:28.069055 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:28.069031 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2140ea23-2136-478a-b9d3-842a0898d794-model-cache" pod "2140ea23-2136-478a-b9d3-842a0898d794" (UID: "2140ea23-2136-478a-b9d3-842a0898d794"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:13:28.069220 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:28.069195 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2140ea23-2136-478a-b9d3-842a0898d794-home" pod "2140ea23-2136-478a-b9d3-842a0898d794" (UID: "2140ea23-2136-478a-b9d3-842a0898d794"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:13:28.069527 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:28.069492 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2140ea23-2136-478a-b9d3-842a0898d794-tmp-dir" pod "2140ea23-2136-478a-b9d3-842a0898d794" (UID: "2140ea23-2136-478a-b9d3-842a0898d794"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:13:28.069643 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:28.069625 5935 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2140ea23-2136-478a-b9d3-842a0898d794-model-cache\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:13:28.069712 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:28.069648 5935 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2140ea23-2136-478a-b9d3-842a0898d794-tmp-dir\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:13:28.069712 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:28.069664 5935 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2140ea23-2136-478a-b9d3-842a0898d794-home\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:13:28.071100 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:28.071071 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2140ea23-2136-478a-b9d3-842a0898d794-tls-certs" pod "2140ea23-2136-478a-b9d3-842a0898d794" (UID: "2140ea23-2136-478a-b9d3-842a0898d794"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:13:28.071100 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:28.071092 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2140ea23-2136-478a-b9d3-842a0898d794-dshm" pod "2140ea23-2136-478a-b9d3-842a0898d794" (UID: "2140ea23-2136-478a-b9d3-842a0898d794"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:13:28.071682 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:28.071656 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2140ea23-2136-478a-b9d3-842a0898d794-kube-api-access-c6sng" pod "2140ea23-2136-478a-b9d3-842a0898d794" (UID: "2140ea23-2136-478a-b9d3-842a0898d794"). InnerVolumeSpecName "kube-api-access-c6sng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:13:28.171138 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:28.171109 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6sng\" (UniqueName: \"kubernetes.io/projected/2140ea23-2136-478a-b9d3-842a0898d794-kube-api-access-c6sng\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:13:28.171138 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:28.171138 5935 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2140ea23-2136-478a-b9d3-842a0898d794-dshm\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:13:28.171375 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:28.171152 5935 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2140ea23-2136-478a-b9d3-842a0898d794-tls-certs\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:13:28.860722 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:28.860691 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-638f19/praxis-route-638f19-kserve-79cd6d56c-2gqpt" Nov 01 20:13:28.861166 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:28.860716 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-638f19/praxis-route-638f19-kserve-79cd6d56c-2gqpt" event={"ID":"2140ea23-2136-478a-b9d3-842a0898d794","Type":"ContainerDied","Data":"5d6fd998c9da5d971696796c4c5fe3bf07ed13dec572fad2947385972c43a864"} Nov 01 20:13:28.861166 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:28.860776 5935 scope.go:122] "RemoveContainer" containerID="aa471da2c839266c20e8e61718d6b4791edc45e0101a1ba02ee7a53f37de4a3d" Nov 01 20:13:28.879171 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:28.879144 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-638f19/praxis-route-638f19-kserve-79cd6d56c-2gqpt"] Nov 01 20:13:28.881926 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:28.881905 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-638f19/praxis-route-638f19-kserve-79cd6d56c-2gqpt"] Nov 01 20:13:29.839955 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:29.839924 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0ebb0cff-6d8669c646-8z9dg" Nov 01 20:13:30.354091 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:30.354058 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2140ea23-2136-478a-b9d3-842a0898d794" path="/var/lib/kubelet/pods/2140ea23-2136-478a-b9d3-842a0898d794/volumes" Nov 01 20:13:33.071678 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:33.071645 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-22g4n"] Nov 01 20:13:33.072268 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:33.072252 5935 cpu_manager.go:414] "containerMap: removing container" podUID="2140ea23-2136-478a-b9d3-842a0898d794" containerName="main" Nov 01 20:13:33.072339 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:33.072268 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2140ea23-2136-478a-b9d3-842a0898d794" containerName="main" Nov 01 20:13:33.072382 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:33.072375 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="2140ea23-2136-478a-b9d3-842a0898d794" containerName="main" Nov 01 20:13:33.076264 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:33.076248 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-22g4n" Nov 01 20:13:33.084791 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:33.084751 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-zbwtw\"" Nov 01 20:13:33.084953 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:33.084850 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 01 20:13:33.084953 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:33.084889 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 01 20:13:33.087678 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:33.087651 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 01 20:13:33.094365 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:33.094341 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-22g4n"] Nov 01 20:13:33.135344 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:33.134229 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/52680409-4495-4000-ade4-4498da85c23b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-22g4n\" (UID: \"52680409-4495-4000-ade4-4498da85c23b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-22g4n" Nov 01 20:13:33.135344 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:33.134359 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhh6r\" (UniqueName: \"kubernetes.io/projected/52680409-4495-4000-ade4-4498da85c23b-kube-api-access-zhh6r\") pod \"istiod-openshift-gateway-7ddd9cfdc8-22g4n\" (UID: \"52680409-4495-4000-ade4-4498da85c23b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-22g4n" Nov 01 20:13:33.135344 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:33.134415 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/52680409-4495-4000-ade4-4498da85c23b-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-22g4n\" (UID: \"52680409-4495-4000-ade4-4498da85c23b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-22g4n" Nov 01 20:13:33.135344 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:33.134455 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/52680409-4495-4000-ade4-4498da85c23b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-22g4n\" (UID: \"52680409-4495-4000-ade4-4498da85c23b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-22g4n" Nov 01 20:13:33.135344 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:33.134517 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/52680409-4495-4000-ade4-4498da85c23b-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-22g4n\" (UID: \"52680409-4495-4000-ade4-4498da85c23b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-22g4n" Nov 01 20:13:33.135344 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:33.134614 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/52680409-4495-4000-ade4-4498da85c23b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-22g4n\" (UID: \"52680409-4495-4000-ade4-4498da85c23b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-22g4n" Nov 01 20:13:33.135344 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:33.134672 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52680409-4495-4000-ade4-4498da85c23b-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-22g4n\" (UID: \"52680409-4495-4000-ade4-4498da85c23b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-22g4n" Nov 01 20:13:33.235956 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:33.235923 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhh6r\" (UniqueName: \"kubernetes.io/projected/52680409-4495-4000-ade4-4498da85c23b-kube-api-access-zhh6r\") pod \"istiod-openshift-gateway-7ddd9cfdc8-22g4n\" (UID: \"52680409-4495-4000-ade4-4498da85c23b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-22g4n" Nov 01 20:13:33.236167 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:33.235980 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/52680409-4495-4000-ade4-4498da85c23b-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-22g4n\" (UID: \"52680409-4495-4000-ade4-4498da85c23b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-22g4n" Nov 01 20:13:33.236167 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:33.236015 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/52680409-4495-4000-ade4-4498da85c23b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-22g4n\" (UID: \"52680409-4495-4000-ade4-4498da85c23b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-22g4n" Nov 01 20:13:33.236167 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:33.236064 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/52680409-4495-4000-ade4-4498da85c23b-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-22g4n\" (UID: \"52680409-4495-4000-ade4-4498da85c23b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-22g4n" Nov 01 20:13:33.236167 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:33.236138 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/52680409-4495-4000-ade4-4498da85c23b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-22g4n\" (UID: \"52680409-4495-4000-ade4-4498da85c23b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-22g4n" Nov 01 20:13:33.236414 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:33.236188 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52680409-4495-4000-ade4-4498da85c23b-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-22g4n\" (UID: \"52680409-4495-4000-ade4-4498da85c23b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-22g4n" Nov 01 20:13:33.236414 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:33.236256 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/52680409-4495-4000-ade4-4498da85c23b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-22g4n\" (UID: \"52680409-4495-4000-ade4-4498da85c23b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-22g4n" Nov 01 20:13:33.237593 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:33.237562 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/52680409-4495-4000-ade4-4498da85c23b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-22g4n\" (UID: \"52680409-4495-4000-ade4-4498da85c23b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-22g4n" Nov 01 20:13:33.238834 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:33.238806 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/52680409-4495-4000-ade4-4498da85c23b-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-22g4n\" (UID: \"52680409-4495-4000-ade4-4498da85c23b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-22g4n" Nov 01 20:13:33.239137 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:33.239113 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/52680409-4495-4000-ade4-4498da85c23b-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-22g4n\" (UID: \"52680409-4495-4000-ade4-4498da85c23b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-22g4n" Nov 01 20:13:33.239229 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:33.239121 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/52680409-4495-4000-ade4-4498da85c23b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-22g4n\" (UID: \"52680409-4495-4000-ade4-4498da85c23b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-22g4n" Nov 01 20:13:33.239483 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:33.239461 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/52680409-4495-4000-ade4-4498da85c23b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-22g4n\" (UID: \"52680409-4495-4000-ade4-4498da85c23b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-22g4n" Nov 01 20:13:33.244808 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:33.244782 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhh6r\" (UniqueName: \"kubernetes.io/projected/52680409-4495-4000-ade4-4498da85c23b-kube-api-access-zhh6r\") pod \"istiod-openshift-gateway-7ddd9cfdc8-22g4n\" (UID: \"52680409-4495-4000-ade4-4498da85c23b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-22g4n" Nov 01 20:13:33.245108 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:33.245093 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52680409-4495-4000-ade4-4498da85c23b-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-22g4n\" (UID: \"52680409-4495-4000-ade4-4498da85c23b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-22g4n" Nov 01 20:13:33.410980 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:33.410897 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-22g4n" Nov 01 20:13:33.689916 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:33.689886 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-22g4n"] Nov 01 20:13:33.690589 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:13:33.690562 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52680409_4495_4000_ade4_4498da85c23b.slice/crio-b96980928473bf297ebf88c95152acc0beaeb13b491bd6fd16e73bd6b45ce35e WatchSource:0}: Error finding container b96980928473bf297ebf88c95152acc0beaeb13b491bd6fd16e73bd6b45ce35e: Status 404 returned error can't find the container with id b96980928473bf297ebf88c95152acc0beaeb13b491bd6fd16e73bd6b45ce35e Nov 01 20:13:33.900719 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:33.900685 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-22g4n" event={"ID":"52680409-4495-4000-ade4-4498da85c23b","Type":"ContainerStarted","Data":"b96980928473bf297ebf88c95152acc0beaeb13b491bd6fd16e73bd6b45ce35e"} Nov 01 20:13:34.166521 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:34.166488 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxpr5s"] Nov 01 20:13:35.853969 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:13:35.853926 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:13:36.866302 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:36.866257 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:13:36.866583 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:36.866336 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:13:37.937907 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:37.937872 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-22g4n" event={"ID":"52680409-4495-4000-ade4-4498da85c23b","Type":"ContainerStarted","Data":"022a2f151e43c1345eca22d401d2e1a961833cf4e03cb450b926cbaa5b8afffb"} Nov 01 20:13:37.938456 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:37.938111 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-22g4n" Nov 01 20:13:37.940248 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:37.940220 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-22g4n" Nov 01 20:13:37.960865 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:37.960813 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-22g4n" podStartSLOduration=1.7885280890000002 podStartE2EDuration="4.960796958s" totalImagesPullingTime="3.172268869s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:13:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:13:37.956409746 +0000 UTC m=+863.729570258" watchObservedRunningTime="2026-11-01 20:13:37.960796958 +0000 UTC m=+863.733957472" Nov 01 20:13:39.167628 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:39.167558 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxpr5s" podUID="1420bd3e-aa71-43bb-98a9-00dbfc4812b1" containerName="payload-processing" containerID="cri-o://c47c77e5362407555e3a9c9592b6227f4e3b9eaace029ace5458aed73c7ee9eb" gracePeriod=25 Nov 01 20:13:39.435332 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:39.435246 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxpr5s" Nov 01 20:13:39.519740 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:39.519706 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1420bd3e-aa71-43bb-98a9-00dbfc4812b1-kube-api-access-htwx4\" (UniqueName: \"kubernetes.io/projected/1420bd3e-aa71-43bb-98a9-00dbfc4812b1-kube-api-access-htwx4\") pod \"1420bd3e-aa71-43bb-98a9-00dbfc4812b1\" (UID: \"1420bd3e-aa71-43bb-98a9-00dbfc4812b1\") " Nov 01 20:13:39.519907 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:39.519800 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1420bd3e-aa71-43bb-98a9-00dbfc4812b1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1420bd3e-aa71-43bb-98a9-00dbfc4812b1-plugins-config-volume\") pod \"1420bd3e-aa71-43bb-98a9-00dbfc4812b1\" (UID: \"1420bd3e-aa71-43bb-98a9-00dbfc4812b1\") " Nov 01 20:13:39.519907 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:39.519844 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1420bd3e-aa71-43bb-98a9-00dbfc4812b1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1420bd3e-aa71-43bb-98a9-00dbfc4812b1-routing-overlay-volume\") pod \"1420bd3e-aa71-43bb-98a9-00dbfc4812b1\" (UID: \"1420bd3e-aa71-43bb-98a9-00dbfc4812b1\") " Nov 01 20:13:39.519907 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:39.519884 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1420bd3e-aa71-43bb-98a9-00dbfc4812b1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1420bd3e-aa71-43bb-98a9-00dbfc4812b1-provider-credentials\") pod \"1420bd3e-aa71-43bb-98a9-00dbfc4812b1\" (UID: \"1420bd3e-aa71-43bb-98a9-00dbfc4812b1\") " Nov 01 20:13:39.520244 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:39.520217 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1420bd3e-aa71-43bb-98a9-00dbfc4812b1-routing-overlay-volume" pod "1420bd3e-aa71-43bb-98a9-00dbfc4812b1" (UID: "1420bd3e-aa71-43bb-98a9-00dbfc4812b1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:13:39.520352 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:39.520248 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1420bd3e-aa71-43bb-98a9-00dbfc4812b1-plugins-config-volume" pod "1420bd3e-aa71-43bb-98a9-00dbfc4812b1" (UID: "1420bd3e-aa71-43bb-98a9-00dbfc4812b1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:13:39.520445 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:39.520426 5935 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1420bd3e-aa71-43bb-98a9-00dbfc4812b1-plugins-config-volume\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:13:39.520510 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:39.520451 5935 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1420bd3e-aa71-43bb-98a9-00dbfc4812b1-routing-overlay-volume\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:13:39.521941 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:39.521920 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1420bd3e-aa71-43bb-98a9-00dbfc4812b1-kube-api-access-htwx4" pod "1420bd3e-aa71-43bb-98a9-00dbfc4812b1" (UID: "1420bd3e-aa71-43bb-98a9-00dbfc4812b1"). InnerVolumeSpecName "kube-api-access-htwx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:13:39.522024 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:39.521990 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1420bd3e-aa71-43bb-98a9-00dbfc4812b1-provider-credentials" pod "1420bd3e-aa71-43bb-98a9-00dbfc4812b1" (UID: "1420bd3e-aa71-43bb-98a9-00dbfc4812b1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:13:39.621374 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:39.621338 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htwx4\" (UniqueName: \"kubernetes.io/projected/1420bd3e-aa71-43bb-98a9-00dbfc4812b1-kube-api-access-htwx4\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:13:39.621374 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:39.621371 5935 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1420bd3e-aa71-43bb-98a9-00dbfc4812b1-provider-credentials\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:13:39.957505 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:39.957465 5935 generic.go:356] "Generic (PLEG): container finished" podID="1420bd3e-aa71-43bb-98a9-00dbfc4812b1" containerID="c47c77e5362407555e3a9c9592b6227f4e3b9eaace029ace5458aed73c7ee9eb" exitCode=0 Nov 01 20:13:39.958373 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:39.958351 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxpr5s" Nov 01 20:13:39.966392 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:39.966349 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxpr5s" event={"ID":"1420bd3e-aa71-43bb-98a9-00dbfc4812b1","Type":"ContainerDied","Data":"c47c77e5362407555e3a9c9592b6227f4e3b9eaace029ace5458aed73c7ee9eb"} Nov 01 20:13:39.966523 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:39.966410 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxpr5s" event={"ID":"1420bd3e-aa71-43bb-98a9-00dbfc4812b1","Type":"ContainerDied","Data":"a39c303d61fff772ec0ff89597e181d641deb23f9abe2934a0272329b5723897"} Nov 01 20:13:39.966523 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:39.966458 5935 scope.go:122] "RemoveContainer" containerID="c47c77e5362407555e3a9c9592b6227f4e3b9eaace029ace5458aed73c7ee9eb" Nov 01 20:13:39.977594 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:39.977572 5935 scope.go:122] "RemoveContainer" containerID="c47c77e5362407555e3a9c9592b6227f4e3b9eaace029ace5458aed73c7ee9eb" Nov 01 20:13:39.977860 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:13:39.977842 5935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c47c77e5362407555e3a9c9592b6227f4e3b9eaace029ace5458aed73c7ee9eb\": container with ID starting with c47c77e5362407555e3a9c9592b6227f4e3b9eaace029ace5458aed73c7ee9eb not found: ID does not exist" containerID="c47c77e5362407555e3a9c9592b6227f4e3b9eaace029ace5458aed73c7ee9eb" Nov 01 20:13:39.977906 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:39.977868 5935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c47c77e5362407555e3a9c9592b6227f4e3b9eaace029ace5458aed73c7ee9eb"} err="failed to get container status \"c47c77e5362407555e3a9c9592b6227f4e3b9eaace029ace5458aed73c7ee9eb\": rpc error: code = NotFound desc = could not find container \"c47c77e5362407555e3a9c9592b6227f4e3b9eaace029ace5458aed73c7ee9eb\": container with ID starting with c47c77e5362407555e3a9c9592b6227f4e3b9eaace029ace5458aed73c7ee9eb not found: ID does not exist" Nov 01 20:13:39.987173 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:39.987151 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxpr5s"] Nov 01 20:13:39.991187 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:39.991160 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-5370bb/payload-processing-external-model-e2e-extmod-b-5370bb-5b8dxpr5s"] Nov 01 20:13:40.359971 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:40.359882 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1420bd3e-aa71-43bb-98a9-00dbfc4812b1" path="/var/lib/kubelet/pods/1420bd3e-aa71-43bb-98a9-00dbfc4812b1/volumes" Nov 01 20:13:43.494971 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:43.494933 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw"] Nov 01 20:13:43.495607 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:43.495592 5935 cpu_manager.go:414] "containerMap: removing container" podUID="1420bd3e-aa71-43bb-98a9-00dbfc4812b1" containerName="payload-processing" Nov 01 20:13:43.495607 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:43.495608 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1420bd3e-aa71-43bb-98a9-00dbfc4812b1" containerName="payload-processing" Nov 01 20:13:43.495743 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:43.495733 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="1420bd3e-aa71-43bb-98a9-00dbfc4812b1" containerName="payload-processing" Nov 01 20:13:43.500591 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:43.500572 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw" Nov 01 20:13:43.508175 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:43.508154 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-5370bb-openshift-default-dockercfg-m26p7\"" Nov 01 20:13:43.512596 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:43.512571 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw"] Nov 01 20:13:43.571549 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:43.571514 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab7a8222-c059-4ecd-8187-fb711ac219c4-istio-envoy\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw\" (UID: \"ab7a8222-c059-4ecd-8187-fb711ac219c4\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw" Nov 01 20:13:43.571755 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:43.571559 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab7a8222-c059-4ecd-8187-fb711ac219c4-credential-socket\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw\" (UID: \"ab7a8222-c059-4ecd-8187-fb711ac219c4\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw" Nov 01 20:13:43.571755 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:43.571589 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab7a8222-c059-4ecd-8187-fb711ac219c4-istio-data\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw\" (UID: \"ab7a8222-c059-4ecd-8187-fb711ac219c4\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw" Nov 01 20:13:43.571755 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:43.571708 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm587\" (UniqueName: \"kubernetes.io/projected/ab7a8222-c059-4ecd-8187-fb711ac219c4-kube-api-access-bm587\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw\" (UID: \"ab7a8222-c059-4ecd-8187-fb711ac219c4\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw" Nov 01 20:13:43.571927 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:43.571758 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab7a8222-c059-4ecd-8187-fb711ac219c4-istio-podinfo\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw\" (UID: \"ab7a8222-c059-4ecd-8187-fb711ac219c4\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw" Nov 01 20:13:43.571927 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:43.571838 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab7a8222-c059-4ecd-8187-fb711ac219c4-istio-token\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw\" (UID: \"ab7a8222-c059-4ecd-8187-fb711ac219c4\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw" Nov 01 20:13:43.571927 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:43.571893 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab7a8222-c059-4ecd-8187-fb711ac219c4-workload-certs\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw\" (UID: \"ab7a8222-c059-4ecd-8187-fb711ac219c4\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw" Nov 01 20:13:43.572093 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:43.571935 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab7a8222-c059-4ecd-8187-fb711ac219c4-istiod-ca-cert\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw\" (UID: \"ab7a8222-c059-4ecd-8187-fb711ac219c4\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw" Nov 01 20:13:43.572093 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:43.572009 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab7a8222-c059-4ecd-8187-fb711ac219c4-workload-socket\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw\" (UID: \"ab7a8222-c059-4ecd-8187-fb711ac219c4\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw" Nov 01 20:13:43.672760 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:43.672724 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab7a8222-c059-4ecd-8187-fb711ac219c4-istio-envoy\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw\" (UID: \"ab7a8222-c059-4ecd-8187-fb711ac219c4\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw" Nov 01 20:13:43.672760 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:43.672766 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab7a8222-c059-4ecd-8187-fb711ac219c4-credential-socket\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw\" (UID: \"ab7a8222-c059-4ecd-8187-fb711ac219c4\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw" Nov 01 20:13:43.673002 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:43.672789 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab7a8222-c059-4ecd-8187-fb711ac219c4-istio-data\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw\" (UID: \"ab7a8222-c059-4ecd-8187-fb711ac219c4\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw" Nov 01 20:13:43.673002 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:43.672821 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm587\" (UniqueName: \"kubernetes.io/projected/ab7a8222-c059-4ecd-8187-fb711ac219c4-kube-api-access-bm587\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw\" (UID: \"ab7a8222-c059-4ecd-8187-fb711ac219c4\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw" Nov 01 20:13:43.673002 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:43.672857 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab7a8222-c059-4ecd-8187-fb711ac219c4-istio-podinfo\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw\" (UID: \"ab7a8222-c059-4ecd-8187-fb711ac219c4\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw" Nov 01 20:13:43.673002 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:43.672942 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab7a8222-c059-4ecd-8187-fb711ac219c4-istio-token\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw\" (UID: \"ab7a8222-c059-4ecd-8187-fb711ac219c4\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw" Nov 01 20:13:43.673214 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:43.673043 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab7a8222-c059-4ecd-8187-fb711ac219c4-workload-certs\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw\" (UID: \"ab7a8222-c059-4ecd-8187-fb711ac219c4\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw" Nov 01 20:13:43.673214 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:43.673110 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab7a8222-c059-4ecd-8187-fb711ac219c4-istiod-ca-cert\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw\" (UID: \"ab7a8222-c059-4ecd-8187-fb711ac219c4\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw" Nov 01 20:13:43.673214 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:43.673154 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab7a8222-c059-4ecd-8187-fb711ac219c4-workload-socket\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw\" (UID: \"ab7a8222-c059-4ecd-8187-fb711ac219c4\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw" Nov 01 20:13:43.673400 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:43.673233 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab7a8222-c059-4ecd-8187-fb711ac219c4-credential-socket\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw\" (UID: \"ab7a8222-c059-4ecd-8187-fb711ac219c4\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw" Nov 01 20:13:43.673400 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:43.673339 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab7a8222-c059-4ecd-8187-fb711ac219c4-istio-data\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw\" (UID: \"ab7a8222-c059-4ecd-8187-fb711ac219c4\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw" Nov 01 20:13:43.673508 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:43.673485 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab7a8222-c059-4ecd-8187-fb711ac219c4-workload-certs\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw\" (UID: \"ab7a8222-c059-4ecd-8187-fb711ac219c4\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw" Nov 01 20:13:43.673630 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:43.673610 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab7a8222-c059-4ecd-8187-fb711ac219c4-workload-socket\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw\" (UID: \"ab7a8222-c059-4ecd-8187-fb711ac219c4\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw" Nov 01 20:13:43.674854 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:43.674049 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab7a8222-c059-4ecd-8187-fb711ac219c4-istiod-ca-cert\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw\" (UID: \"ab7a8222-c059-4ecd-8187-fb711ac219c4\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw" Nov 01 20:13:43.675639 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:43.675616 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab7a8222-c059-4ecd-8187-fb711ac219c4-istio-podinfo\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw\" (UID: \"ab7a8222-c059-4ecd-8187-fb711ac219c4\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw" Nov 01 20:13:43.681605 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:43.681580 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab7a8222-c059-4ecd-8187-fb711ac219c4-istio-envoy\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw\" (UID: \"ab7a8222-c059-4ecd-8187-fb711ac219c4\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw" Nov 01 20:13:43.682852 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:43.682830 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm587\" (UniqueName: \"kubernetes.io/projected/ab7a8222-c059-4ecd-8187-fb711ac219c4-kube-api-access-bm587\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw\" (UID: \"ab7a8222-c059-4ecd-8187-fb711ac219c4\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw" Nov 01 20:13:43.684184 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:43.684161 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab7a8222-c059-4ecd-8187-fb711ac219c4-istio-token\") pod \"e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw\" (UID: \"ab7a8222-c059-4ecd-8187-fb711ac219c4\") " pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw" Nov 01 20:13:43.815237 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:43.815152 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw" Nov 01 20:13:44.115109 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:44.115078 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw"] Nov 01 20:13:44.116272 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:13:44.116245 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab7a8222_c059_4ecd_8187_fb711ac219c4.slice/crio-a31c74e6682159595a4f2b25d76b7df60f53fb4674779f9878b3da1605f932ff WatchSource:0}: Error finding container a31c74e6682159595a4f2b25d76b7df60f53fb4674779f9878b3da1605f932ff: Status 404 returned error can't find the container with id a31c74e6682159595a4f2b25d76b7df60f53fb4674779f9878b3da1605f932ff Nov 01 20:13:44.118675 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:44.118639 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:13:44.118782 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:44.118716 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:13:44.118782 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:44.118764 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:13:44.992880 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:44.992840 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw" event={"ID":"ab7a8222-c059-4ecd-8187-fb711ac219c4","Type":"ContainerStarted","Data":"088a4b4d9b3a3b0e63bd1eda3077fc56d3011c3e660c3611f4dbedee35a0e1bb"} Nov 01 20:13:44.993419 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:44.992937 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw" event={"ID":"ab7a8222-c059-4ecd-8187-fb711ac219c4","Type":"ContainerStarted","Data":"a31c74e6682159595a4f2b25d76b7df60f53fb4674779f9878b3da1605f932ff"} Nov 01 20:13:45.015218 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:45.015165 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw" podStartSLOduration=2.015153346 podStartE2EDuration="2.015153346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:13:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:13:45.014016874 +0000 UTC m=+870.787177389" watchObservedRunningTime="2026-11-01 20:13:45.015153346 +0000 UTC m=+870.788313857" Nov 01 20:13:45.816143 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:45.816104 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw" Nov 01 20:13:45.817420 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:45.817388 5935 patch_prober.go:29] interesting pod/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 01 20:13:45.817543 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:45.817439 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw" podUID="ab7a8222-c059-4ecd-8187-fb711ac219c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 01 20:13:46.815666 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:46.815634 5935 patch_prober.go:29] interesting pod/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 01 20:13:46.816067 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:46.815680 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw" podUID="ab7a8222-c059-4ecd-8187-fb711ac219c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 01 20:13:47.816095 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:47.816062 5935 patch_prober.go:29] interesting pod/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 01 20:13:47.816507 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:47.816123 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw" podUID="ab7a8222-c059-4ecd-8187-fb711ac219c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 01 20:13:48.815981 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:48.815945 5935 patch_prober.go:29] interesting pod/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 01 20:13:48.816154 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:48.815999 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw" podUID="ab7a8222-c059-4ecd-8187-fb711ac219c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 01 20:13:49.527390 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:49.527355 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0ebb0cff-7bcc55b8cf-89s86"] Nov 01 20:13:49.816155 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:49.816065 5935 patch_prober.go:29] interesting pod/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" start-of-body= Nov 01 20:13:49.816750 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:49.816713 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw" podUID="ab7a8222-c059-4ecd-8187-fb711ac219c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.79:15021/healthz/ready\": dial tcp 10.133.0.79:15021: connect: connection refused" Nov 01 20:13:50.819509 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:50.819482 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw" Nov 01 20:13:50.819892 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:50.819741 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw" Nov 01 20:13:50.820494 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:50.820475 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw" Nov 01 20:13:54.379598 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:54.379560 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0ebb0cff-6d8669c646-8z9dg"] Nov 01 20:13:54.380099 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:54.379773 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0ebb0cff-6d8669c646-8z9dg" podUID="11752a2a-c301-4a45-89d8-470fcc7332f4" containerName="maas-api" containerID="cri-o://e0224df5dab88cccf2a3259bbd0d3c9cf89c8c91b3fe9d5c61f5820eed3ffc09" gracePeriod=30 Nov 01 20:13:54.527999 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:54.527798 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-0ebb0cff-7bcc55b8cf-89s86" podUID="bda5c2cd-0ec3-4b3f-ad58-a51a91c81245" containerName="payload-processing" containerID="cri-o://3e81b93475fcd03377eda8706dd1df7496e6a022e61d436aebb78b377bb7efbe" gracePeriod=25 Nov 01 20:13:54.775468 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:54.775441 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0ebb0cff-6d8669c646-8z9dg" Nov 01 20:13:54.845245 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:54.845210 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11752a2a-c301-4a45-89d8-470fcc7332f4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11752a2a-c301-4a45-89d8-470fcc7332f4-openshift-service-ca\") pod \"11752a2a-c301-4a45-89d8-470fcc7332f4\" (UID: \"11752a2a-c301-4a45-89d8-470fcc7332f4\") " Nov 01 20:13:54.845452 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:54.845281 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11752a2a-c301-4a45-89d8-470fcc7332f4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11752a2a-c301-4a45-89d8-470fcc7332f4-maas-api-tls\") pod \"11752a2a-c301-4a45-89d8-470fcc7332f4\" (UID: \"11752a2a-c301-4a45-89d8-470fcc7332f4\") " Nov 01 20:13:54.845658 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:54.845630 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11752a2a-c301-4a45-89d8-470fcc7332f4-openshift-service-ca" pod "11752a2a-c301-4a45-89d8-470fcc7332f4" (UID: "11752a2a-c301-4a45-89d8-470fcc7332f4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:13:54.845859 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:54.845823 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11752a2a-c301-4a45-89d8-470fcc7332f4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/11752a2a-c301-4a45-89d8-470fcc7332f4-metrics-certs\") pod \"11752a2a-c301-4a45-89d8-470fcc7332f4\" (UID: \"11752a2a-c301-4a45-89d8-470fcc7332f4\") " Nov 01 20:13:54.846051 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:54.845933 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11752a2a-c301-4a45-89d8-470fcc7332f4-kube-api-access-4cxhb\" (UniqueName: \"kubernetes.io/projected/11752a2a-c301-4a45-89d8-470fcc7332f4-kube-api-access-4cxhb\") pod \"11752a2a-c301-4a45-89d8-470fcc7332f4\" (UID: \"11752a2a-c301-4a45-89d8-470fcc7332f4\") " Nov 01 20:13:54.846506 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:54.846487 5935 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/11752a2a-c301-4a45-89d8-470fcc7332f4-openshift-service-ca\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:13:54.848164 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:54.848143 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11752a2a-c301-4a45-89d8-470fcc7332f4-maas-api-tls" pod "11752a2a-c301-4a45-89d8-470fcc7332f4" (UID: "11752a2a-c301-4a45-89d8-470fcc7332f4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:13:54.848514 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:54.848491 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11752a2a-c301-4a45-89d8-470fcc7332f4-kube-api-access-4cxhb" pod "11752a2a-c301-4a45-89d8-470fcc7332f4" (UID: "11752a2a-c301-4a45-89d8-470fcc7332f4"). InnerVolumeSpecName "kube-api-access-4cxhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:13:54.850051 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:54.850023 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11752a2a-c301-4a45-89d8-470fcc7332f4-metrics-certs" pod "11752a2a-c301-4a45-89d8-470fcc7332f4" (UID: "11752a2a-c301-4a45-89d8-470fcc7332f4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:13:54.889203 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:54.889177 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0ebb0cff-7bcc55b8cf-89s86" Nov 01 20:13:54.948801 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:54.947545 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bda5c2cd-0ec3-4b3f-ad58-a51a91c81245-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bda5c2cd-0ec3-4b3f-ad58-a51a91c81245-plugins-config-volume\") pod \"bda5c2cd-0ec3-4b3f-ad58-a51a91c81245\" (UID: \"bda5c2cd-0ec3-4b3f-ad58-a51a91c81245\") " Nov 01 20:13:54.948801 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:54.947849 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bda5c2cd-0ec3-4b3f-ad58-a51a91c81245-kube-api-access-njdf8\" (UniqueName: \"kubernetes.io/projected/bda5c2cd-0ec3-4b3f-ad58-a51a91c81245-kube-api-access-njdf8\") pod \"bda5c2cd-0ec3-4b3f-ad58-a51a91c81245\" (UID: \"bda5c2cd-0ec3-4b3f-ad58-a51a91c81245\") " Nov 01 20:13:54.948801 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:54.948388 5935 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/11752a2a-c301-4a45-89d8-470fcc7332f4-metrics-certs\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:13:54.948801 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:54.948411 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cxhb\" (UniqueName: \"kubernetes.io/projected/11752a2a-c301-4a45-89d8-470fcc7332f4-kube-api-access-4cxhb\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:13:54.948801 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:54.948428 5935 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/11752a2a-c301-4a45-89d8-470fcc7332f4-maas-api-tls\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:13:54.949512 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:54.949111 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bda5c2cd-0ec3-4b3f-ad58-a51a91c81245-plugins-config-volume" pod "bda5c2cd-0ec3-4b3f-ad58-a51a91c81245" (UID: "bda5c2cd-0ec3-4b3f-ad58-a51a91c81245"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:13:54.952730 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:54.952699 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bda5c2cd-0ec3-4b3f-ad58-a51a91c81245-kube-api-access-njdf8" pod "bda5c2cd-0ec3-4b3f-ad58-a51a91c81245" (UID: "bda5c2cd-0ec3-4b3f-ad58-a51a91c81245"). InnerVolumeSpecName "kube-api-access-njdf8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:13:55.050147 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:55.050076 5935 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bda5c2cd-0ec3-4b3f-ad58-a51a91c81245-plugins-config-volume\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:13:55.050147 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:55.050103 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njdf8\" (UniqueName: \"kubernetes.io/projected/bda5c2cd-0ec3-4b3f-ad58-a51a91c81245-kube-api-access-njdf8\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:13:55.061417 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:55.061385 5935 generic.go:356] "Generic (PLEG): container finished" podID="11752a2a-c301-4a45-89d8-470fcc7332f4" containerID="e0224df5dab88cccf2a3259bbd0d3c9cf89c8c91b3fe9d5c61f5820eed3ffc09" exitCode=0 Nov 01 20:13:55.061549 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:55.061434 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0ebb0cff-6d8669c646-8z9dg" Nov 01 20:13:55.061549 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:55.061464 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0ebb0cff-6d8669c646-8z9dg" event={"ID":"11752a2a-c301-4a45-89d8-470fcc7332f4","Type":"ContainerDied","Data":"e0224df5dab88cccf2a3259bbd0d3c9cf89c8c91b3fe9d5c61f5820eed3ffc09"} Nov 01 20:13:55.061549 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:55.061518 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0ebb0cff-6d8669c646-8z9dg" event={"ID":"11752a2a-c301-4a45-89d8-470fcc7332f4","Type":"ContainerDied","Data":"a0767518eed69822e1e9fa7f4931a7c11ded079353b2324fddb51d1998acd913"} Nov 01 20:13:55.061683 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:55.061550 5935 scope.go:122] "RemoveContainer" containerID="e0224df5dab88cccf2a3259bbd0d3c9cf89c8c91b3fe9d5c61f5820eed3ffc09" Nov 01 20:13:55.062861 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:55.062837 5935 generic.go:356] "Generic (PLEG): container finished" podID="bda5c2cd-0ec3-4b3f-ad58-a51a91c81245" containerID="3e81b93475fcd03377eda8706dd1df7496e6a022e61d436aebb78b377bb7efbe" exitCode=0 Nov 01 20:13:55.062985 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:55.062878 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0ebb0cff-7bcc55b8cf-89s86" Nov 01 20:13:55.062985 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:55.062920 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0ebb0cff-7bcc55b8cf-89s86" event={"ID":"bda5c2cd-0ec3-4b3f-ad58-a51a91c81245","Type":"ContainerDied","Data":"3e81b93475fcd03377eda8706dd1df7496e6a022e61d436aebb78b377bb7efbe"} Nov 01 20:13:55.063085 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:55.062986 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0ebb0cff-7bcc55b8cf-89s86" event={"ID":"bda5c2cd-0ec3-4b3f-ad58-a51a91c81245","Type":"ContainerDied","Data":"8a102e071f355ae410e9e1508a09470e46acd3c610240b20f3540350d0ccda56"} Nov 01 20:13:55.077398 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:55.077299 5935 scope.go:122] "RemoveContainer" containerID="e0224df5dab88cccf2a3259bbd0d3c9cf89c8c91b3fe9d5c61f5820eed3ffc09" Nov 01 20:13:55.077857 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:13:55.077726 5935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0224df5dab88cccf2a3259bbd0d3c9cf89c8c91b3fe9d5c61f5820eed3ffc09\": container with ID starting with e0224df5dab88cccf2a3259bbd0d3c9cf89c8c91b3fe9d5c61f5820eed3ffc09 not found: ID does not exist" containerID="e0224df5dab88cccf2a3259bbd0d3c9cf89c8c91b3fe9d5c61f5820eed3ffc09" Nov 01 20:13:55.077857 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:55.077759 5935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0224df5dab88cccf2a3259bbd0d3c9cf89c8c91b3fe9d5c61f5820eed3ffc09"} err="failed to get container status \"e0224df5dab88cccf2a3259bbd0d3c9cf89c8c91b3fe9d5c61f5820eed3ffc09\": rpc error: code = NotFound desc = could not find container \"e0224df5dab88cccf2a3259bbd0d3c9cf89c8c91b3fe9d5c61f5820eed3ffc09\": container with ID starting with e0224df5dab88cccf2a3259bbd0d3c9cf89c8c91b3fe9d5c61f5820eed3ffc09 not found: ID does not exist" Nov 01 20:13:55.077857 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:55.077782 5935 scope.go:122] "RemoveContainer" containerID="3e81b93475fcd03377eda8706dd1df7496e6a022e61d436aebb78b377bb7efbe" Nov 01 20:13:55.148195 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:55.148162 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0ebb0cff-7bcc55b8cf-89s86"] Nov 01 20:13:55.148535 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:55.148509 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0ebb0cff-7bcc55b8cf-89s86"] Nov 01 20:13:55.155657 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:55.155612 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0ebb0cff-6d8669c646-8z9dg"] Nov 01 20:13:55.156616 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:55.156591 5935 scope.go:122] "RemoveContainer" containerID="3e81b93475fcd03377eda8706dd1df7496e6a022e61d436aebb78b377bb7efbe" Nov 01 20:13:55.157052 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:13:55.157023 5935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e81b93475fcd03377eda8706dd1df7496e6a022e61d436aebb78b377bb7efbe\": container with ID starting with 3e81b93475fcd03377eda8706dd1df7496e6a022e61d436aebb78b377bb7efbe not found: ID does not exist" containerID="3e81b93475fcd03377eda8706dd1df7496e6a022e61d436aebb78b377bb7efbe" Nov 01 20:13:55.157144 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:55.157063 5935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e81b93475fcd03377eda8706dd1df7496e6a022e61d436aebb78b377bb7efbe"} err="failed to get container status \"3e81b93475fcd03377eda8706dd1df7496e6a022e61d436aebb78b377bb7efbe\": rpc error: code = NotFound desc = could not find container \"3e81b93475fcd03377eda8706dd1df7496e6a022e61d436aebb78b377bb7efbe\": container with ID starting with 3e81b93475fcd03377eda8706dd1df7496e6a022e61d436aebb78b377bb7efbe not found: ID does not exist" Nov 01 20:13:55.160745 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:55.160724 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0ebb0cff-6d8669c646-8z9dg"] Nov 01 20:13:56.357365 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:56.357328 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11752a2a-c301-4a45-89d8-470fcc7332f4" path="/var/lib/kubelet/pods/11752a2a-c301-4a45-89d8-470fcc7332f4/volumes" Nov 01 20:13:56.357842 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:13:56.357793 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bda5c2cd-0ec3-4b3f-ad58-a51a91c81245" path="/var/lib/kubelet/pods/bda5c2cd-0ec3-4b3f-ad58-a51a91c81245/volumes" Nov 01 20:14:05.848980 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:14:05.848940 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:14:20.399893 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:14:20.399681 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-7f0c86-946796b95-bbvrc"] Nov 01 20:14:20.403000 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:14:20.402968 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-7f0c86-84cd8cd7d9-pfqdn"] Nov 01 20:14:25.400731 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:14:25.400651 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7f0c86-946796b95-bbvrc" podUID="657cdb6c-3591-4340-8737-c15687f3f32b" containerName="payload-pre-processing" containerID="cri-o://f73febf61804860b2af76c50e3f160554df68b3c9df63dd029d855a19409ec18" gracePeriod=25 Nov 01 20:14:25.404234 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:14:25.404206 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-7f0c86-84cd8cd7d9-pfqdn" podUID="e3b6aa7b-cfb2-48b4-a762-e3dd5cbb1f41" containerName="payload-processing" containerID="cri-o://e065ac96545bda94bbc74205a70d2b8f1e2bcfe9ab9fce04ab6148f67019c389" gracePeriod=25 Nov 01 20:14:25.738146 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:14:25.738118 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7f0c86-946796b95-bbvrc" Nov 01 20:14:25.741653 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:14:25.741632 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-7f0c86-84cd8cd7d9-pfqdn" Nov 01 20:14:25.831278 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:14:25.831249 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/657cdb6c-3591-4340-8737-c15687f3f32b-kube-api-access-7b49b\" (UniqueName: \"kubernetes.io/projected/657cdb6c-3591-4340-8737-c15687f3f32b-kube-api-access-7b49b\") pod \"657cdb6c-3591-4340-8737-c15687f3f32b\" (UID: \"657cdb6c-3591-4340-8737-c15687f3f32b\") " Nov 01 20:14:25.831468 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:14:25.831332 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3b6aa7b-cfb2-48b4-a762-e3dd5cbb1f41-kube-api-access-fqbmv\" (UniqueName: \"kubernetes.io/projected/e3b6aa7b-cfb2-48b4-a762-e3dd5cbb1f41-kube-api-access-fqbmv\") pod \"e3b6aa7b-cfb2-48b4-a762-e3dd5cbb1f41\" (UID: \"e3b6aa7b-cfb2-48b4-a762-e3dd5cbb1f41\") " Nov 01 20:14:25.831468 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:14:25.831432 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/657cdb6c-3591-4340-8737-c15687f3f32b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/657cdb6c-3591-4340-8737-c15687f3f32b-plugins-config-volume\") pod \"657cdb6c-3591-4340-8737-c15687f3f32b\" (UID: \"657cdb6c-3591-4340-8737-c15687f3f32b\") " Nov 01 20:14:25.831559 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:14:25.831489 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3b6aa7b-cfb2-48b4-a762-e3dd5cbb1f41-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3b6aa7b-cfb2-48b4-a762-e3dd5cbb1f41-plugins-config-volume\") pod \"e3b6aa7b-cfb2-48b4-a762-e3dd5cbb1f41\" (UID: \"e3b6aa7b-cfb2-48b4-a762-e3dd5cbb1f41\") " Nov 01 20:14:25.831806 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:14:25.831781 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/657cdb6c-3591-4340-8737-c15687f3f32b-plugins-config-volume" pod "657cdb6c-3591-4340-8737-c15687f3f32b" (UID: "657cdb6c-3591-4340-8737-c15687f3f32b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:14:25.831902 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:14:25.831856 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3b6aa7b-cfb2-48b4-a762-e3dd5cbb1f41-plugins-config-volume" pod "e3b6aa7b-cfb2-48b4-a762-e3dd5cbb1f41" (UID: "e3b6aa7b-cfb2-48b4-a762-e3dd5cbb1f41"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:14:25.831945 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:14:25.831904 5935 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/657cdb6c-3591-4340-8737-c15687f3f32b-plugins-config-volume\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:14:25.833383 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:14:25.833363 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3b6aa7b-cfb2-48b4-a762-e3dd5cbb1f41-kube-api-access-fqbmv" pod "e3b6aa7b-cfb2-48b4-a762-e3dd5cbb1f41" (UID: "e3b6aa7b-cfb2-48b4-a762-e3dd5cbb1f41"). InnerVolumeSpecName "kube-api-access-fqbmv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:14:25.833501 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:14:25.833487 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/657cdb6c-3591-4340-8737-c15687f3f32b-kube-api-access-7b49b" pod "657cdb6c-3591-4340-8737-c15687f3f32b" (UID: "657cdb6c-3591-4340-8737-c15687f3f32b"). InnerVolumeSpecName "kube-api-access-7b49b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:14:25.932968 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:14:25.932944 5935 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3b6aa7b-cfb2-48b4-a762-e3dd5cbb1f41-plugins-config-volume\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:14:25.932968 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:14:25.932968 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7b49b\" (UniqueName: \"kubernetes.io/projected/657cdb6c-3591-4340-8737-c15687f3f32b-kube-api-access-7b49b\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:14:25.933117 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:14:25.932978 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqbmv\" (UniqueName: \"kubernetes.io/projected/e3b6aa7b-cfb2-48b4-a762-e3dd5cbb1f41-kube-api-access-fqbmv\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:14:26.272731 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:14:26.272688 5935 generic.go:356] "Generic (PLEG): container finished" podID="e3b6aa7b-cfb2-48b4-a762-e3dd5cbb1f41" containerID="e065ac96545bda94bbc74205a70d2b8f1e2bcfe9ab9fce04ab6148f67019c389" exitCode=0 Nov 01 20:14:26.272731 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:14:26.272734 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-7f0c86-84cd8cd7d9-pfqdn" Nov 01 20:14:26.273097 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:14:26.272743 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-7f0c86-84cd8cd7d9-pfqdn" event={"ID":"e3b6aa7b-cfb2-48b4-a762-e3dd5cbb1f41","Type":"ContainerDied","Data":"e065ac96545bda94bbc74205a70d2b8f1e2bcfe9ab9fce04ab6148f67019c389"} Nov 01 20:14:26.273097 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:14:26.272803 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-7f0c86-84cd8cd7d9-pfqdn" event={"ID":"e3b6aa7b-cfb2-48b4-a762-e3dd5cbb1f41","Type":"ContainerDied","Data":"86b4063c01e650fcdd44cd57441f70aa86201b55b2d446ca8fe833e4e23fffad"} Nov 01 20:14:26.273097 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:14:26.272828 5935 scope.go:122] "RemoveContainer" containerID="e065ac96545bda94bbc74205a70d2b8f1e2bcfe9ab9fce04ab6148f67019c389" Nov 01 20:14:26.274936 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:14:26.274909 5935 generic.go:356] "Generic (PLEG): container finished" podID="657cdb6c-3591-4340-8737-c15687f3f32b" containerID="f73febf61804860b2af76c50e3f160554df68b3c9df63dd029d855a19409ec18" exitCode=0 Nov 01 20:14:26.275073 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:14:26.275057 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7f0c86-946796b95-bbvrc" event={"ID":"657cdb6c-3591-4340-8737-c15687f3f32b","Type":"ContainerDied","Data":"f73febf61804860b2af76c50e3f160554df68b3c9df63dd029d855a19409ec18"} Nov 01 20:14:26.275137 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:14:26.275102 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7f0c86-946796b95-bbvrc" event={"ID":"657cdb6c-3591-4340-8737-c15687f3f32b","Type":"ContainerDied","Data":"63c36b375587bd3aea8ca3113751b84561141d5b9fdaecb9357aa1ea6e8acc43"} Nov 01 20:14:26.275194 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:14:26.275170 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7f0c86-946796b95-bbvrc" Nov 01 20:14:26.286327 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:14:26.286272 5935 scope.go:122] "RemoveContainer" containerID="e065ac96545bda94bbc74205a70d2b8f1e2bcfe9ab9fce04ab6148f67019c389" Nov 01 20:14:26.286653 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:14:26.286619 5935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e065ac96545bda94bbc74205a70d2b8f1e2bcfe9ab9fce04ab6148f67019c389\": container with ID starting with e065ac96545bda94bbc74205a70d2b8f1e2bcfe9ab9fce04ab6148f67019c389 not found: ID does not exist" containerID="e065ac96545bda94bbc74205a70d2b8f1e2bcfe9ab9fce04ab6148f67019c389" Nov 01 20:14:26.286723 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:14:26.286657 5935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e065ac96545bda94bbc74205a70d2b8f1e2bcfe9ab9fce04ab6148f67019c389"} err="failed to get container status \"e065ac96545bda94bbc74205a70d2b8f1e2bcfe9ab9fce04ab6148f67019c389\": rpc error: code = NotFound desc = could not find container \"e065ac96545bda94bbc74205a70d2b8f1e2bcfe9ab9fce04ab6148f67019c389\": container with ID starting with e065ac96545bda94bbc74205a70d2b8f1e2bcfe9ab9fce04ab6148f67019c389 not found: ID does not exist" Nov 01 20:14:26.286723 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:14:26.286680 5935 scope.go:122] "RemoveContainer" containerID="f73febf61804860b2af76c50e3f160554df68b3c9df63dd029d855a19409ec18" Nov 01 20:14:26.297646 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:14:26.297614 5935 scope.go:122] "RemoveContainer" containerID="f73febf61804860b2af76c50e3f160554df68b3c9df63dd029d855a19409ec18" Nov 01 20:14:26.297957 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:14:26.297927 5935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f73febf61804860b2af76c50e3f160554df68b3c9df63dd029d855a19409ec18\": container with ID starting with f73febf61804860b2af76c50e3f160554df68b3c9df63dd029d855a19409ec18 not found: ID does not exist" containerID="f73febf61804860b2af76c50e3f160554df68b3c9df63dd029d855a19409ec18" Nov 01 20:14:26.298042 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:14:26.297964 5935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f73febf61804860b2af76c50e3f160554df68b3c9df63dd029d855a19409ec18"} err="failed to get container status \"f73febf61804860b2af76c50e3f160554df68b3c9df63dd029d855a19409ec18\": rpc error: code = NotFound desc = could not find container \"f73febf61804860b2af76c50e3f160554df68b3c9df63dd029d855a19409ec18\": container with ID starting with f73febf61804860b2af76c50e3f160554df68b3c9df63dd029d855a19409ec18 not found: ID does not exist" Nov 01 20:14:26.303637 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:14:26.303609 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-7f0c86-946796b95-bbvrc"] Nov 01 20:14:26.309757 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:14:26.309736 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-7f0c86-946796b95-bbvrc"] Nov 01 20:14:26.321848 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:14:26.321822 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-7f0c86-84cd8cd7d9-pfqdn"] Nov 01 20:14:26.329306 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:14:26.326790 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-7f0c86-84cd8cd7d9-pfqdn"] Nov 01 20:14:26.390237 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:14:26.388889 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="657cdb6c-3591-4340-8737-c15687f3f32b" path="/var/lib/kubelet/pods/657cdb6c-3591-4340-8737-c15687f3f32b/volumes" Nov 01 20:14:26.390625 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:14:26.390556 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3b6aa7b-cfb2-48b4-a762-e3dd5cbb1f41" path="/var/lib/kubelet/pods/e3b6aa7b-cfb2-48b4-a762-e3dd5cbb1f41/volumes" Nov 01 20:14:35.896567 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:14:35.896524 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:15:00.171782 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:15:00.171740 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-29b57c-29892735-2sz2s"] Nov 01 20:15:00.172650 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:15:00.172622 5935 cpu_manager.go:414] "containerMap: removing container" podUID="bda5c2cd-0ec3-4b3f-ad58-a51a91c81245" containerName="payload-processing" Nov 01 20:15:00.172650 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:15:00.172649 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bda5c2cd-0ec3-4b3f-ad58-a51a91c81245" containerName="payload-processing" Nov 01 20:15:00.172851 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:15:00.172689 5935 cpu_manager.go:414] "containerMap: removing container" podUID="e3b6aa7b-cfb2-48b4-a762-e3dd5cbb1f41" containerName="payload-processing" Nov 01 20:15:00.172851 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:15:00.172698 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3b6aa7b-cfb2-48b4-a762-e3dd5cbb1f41" containerName="payload-processing" Nov 01 20:15:00.172851 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:15:00.172711 5935 cpu_manager.go:414] "containerMap: removing container" podUID="657cdb6c-3591-4340-8737-c15687f3f32b" containerName="payload-pre-processing" Nov 01 20:15:00.172851 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:15:00.172719 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="657cdb6c-3591-4340-8737-c15687f3f32b" containerName="payload-pre-processing" Nov 01 20:15:00.172851 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:15:00.172738 5935 cpu_manager.go:414] "containerMap: removing container" podUID="11752a2a-c301-4a45-89d8-470fcc7332f4" containerName="maas-api" Nov 01 20:15:00.172851 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:15:00.172746 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11752a2a-c301-4a45-89d8-470fcc7332f4" containerName="maas-api" Nov 01 20:15:00.173190 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:15:00.172888 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3b6aa7b-cfb2-48b4-a762-e3dd5cbb1f41" containerName="payload-processing" Nov 01 20:15:00.173190 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:15:00.172900 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="657cdb6c-3591-4340-8737-c15687f3f32b" containerName="payload-pre-processing" Nov 01 20:15:00.173190 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:15:00.172912 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="11752a2a-c301-4a45-89d8-470fcc7332f4" containerName="maas-api" Nov 01 20:15:00.173190 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:15:00.172929 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="bda5c2cd-0ec3-4b3f-ad58-a51a91c81245" containerName="payload-processing" Nov 01 20:15:00.177354 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:15:00.177333 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-29b57c-29892735-2sz2s" Nov 01 20:15:00.182857 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:15:00.182835 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\"" Nov 01 20:15:00.187234 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:15:00.187213 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-29b57c-29892735-2sz2s"] Nov 01 20:15:00.292430 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:15:00.292400 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/beccc923-162c-4600-a5b1-f6735a009583-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-29b57c-29892735-2sz2s\" (UID: \"beccc923-162c-4600-a5b1-f6735a009583\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-29b57c-29892735-2sz2s" Nov 01 20:15:00.292603 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:15:00.292587 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cdh8\" (UniqueName: \"kubernetes.io/projected/beccc923-162c-4600-a5b1-f6735a009583-kube-api-access-9cdh8\") pod \"maas-api-key-cleanup-e2e-extmod-a-29b57c-29892735-2sz2s\" (UID: \"beccc923-162c-4600-a5b1-f6735a009583\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-29b57c-29892735-2sz2s" Nov 01 20:15:00.394218 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:15:00.394188 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cdh8\" (UniqueName: \"kubernetes.io/projected/beccc923-162c-4600-a5b1-f6735a009583-kube-api-access-9cdh8\") pod \"maas-api-key-cleanup-e2e-extmod-a-29b57c-29892735-2sz2s\" (UID: \"beccc923-162c-4600-a5b1-f6735a009583\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-29b57c-29892735-2sz2s" Nov 01 20:15:00.394407 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:15:00.394250 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/beccc923-162c-4600-a5b1-f6735a009583-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-29b57c-29892735-2sz2s\" (UID: \"beccc923-162c-4600-a5b1-f6735a009583\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-29b57c-29892735-2sz2s" Nov 01 20:15:00.394865 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:15:00.394845 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/beccc923-162c-4600-a5b1-f6735a009583-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-29b57c-29892735-2sz2s\" (UID: \"beccc923-162c-4600-a5b1-f6735a009583\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-29b57c-29892735-2sz2s" Nov 01 20:15:00.402140 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:15:00.402118 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cdh8\" (UniqueName: \"kubernetes.io/projected/beccc923-162c-4600-a5b1-f6735a009583-kube-api-access-9cdh8\") pod \"maas-api-key-cleanup-e2e-extmod-a-29b57c-29892735-2sz2s\" (UID: \"beccc923-162c-4600-a5b1-f6735a009583\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-29b57c-29892735-2sz2s" Nov 01 20:15:00.501396 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:15:00.501379 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-29b57c-29892735-2sz2s" Nov 01 20:15:00.777127 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:15:00.777102 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-29b57c-29892735-2sz2s"] Nov 01 20:15:00.779013 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:15:00.778985 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeccc923_162c_4600_a5b1_f6735a009583.slice/crio-a3e29fa233248fa046c9c2760e2a2a0af12f759e52774ed7a1cc6150eb7c2d3a WatchSource:0}: Error finding container a3e29fa233248fa046c9c2760e2a2a0af12f759e52774ed7a1cc6150eb7c2d3a: Status 404 returned error can't find the container with id a3e29fa233248fa046c9c2760e2a2a0af12f759e52774ed7a1cc6150eb7c2d3a Nov 01 20:15:01.562954 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:15:01.562877 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-29b57c-29892735-2sz2s" event={"ID":"beccc923-162c-4600-a5b1-f6735a009583","Type":"ContainerStarted","Data":"a3e29fa233248fa046c9c2760e2a2a0af12f759e52774ed7a1cc6150eb7c2d3a"} Nov 01 20:15:04.587059 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:15:04.587010 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-29b57c-29892735-2sz2s" event={"ID":"beccc923-162c-4600-a5b1-f6735a009583","Type":"ContainerStarted","Data":"f6f048593531350d31227f55b95391a5bdafdfeda28f2b9a31f6cdc9eb04bacb"} Nov 01 20:15:04.602878 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:15:04.602826 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-29b57c-29892735-2sz2s" podStartSLOduration=1.591943254 podStartE2EDuration="4.602812771s" totalImagesPullingTime="3.010869517s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:15:04.601211445 +0000 UTC m=+950.374371951" watchObservedRunningTime="2026-11-01 20:15:04.602812771 +0000 UTC m=+950.375973283" Nov 01 20:15:05.595138 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:15:05.595102 5935 generic.go:356] "Generic (PLEG): container finished" podID="beccc923-162c-4600-a5b1-f6735a009583" containerID="f6f048593531350d31227f55b95391a5bdafdfeda28f2b9a31f6cdc9eb04bacb" exitCode=60 Nov 01 20:15:05.595591 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:15:05.595178 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-29b57c-29892735-2sz2s" event={"ID":"beccc923-162c-4600-a5b1-f6735a009583","Type":"ContainerDied","Data":"f6f048593531350d31227f55b95391a5bdafdfeda28f2b9a31f6cdc9eb04bacb"} Nov 01 20:15:05.595591 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:15:05.595421 5935 scope.go:122] "RemoveContainer" containerID="f6f048593531350d31227f55b95391a5bdafdfeda28f2b9a31f6cdc9eb04bacb" Nov 01 20:15:05.845616 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:15:05.845536 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:15:06.603484 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:15:06.603450 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-29b57c-29892735-2sz2s" event={"ID":"beccc923-162c-4600-a5b1-f6735a009583","Type":"ContainerStarted","Data":"b12a5781c0d34b707edcc40c49556e5b30a195af0e7aa298080e2b1873416ae6"} Nov 01 20:15:07.611439 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:15:07.611398 5935 generic.go:356] "Generic (PLEG): container finished" podID="beccc923-162c-4600-a5b1-f6735a009583" containerID="b12a5781c0d34b707edcc40c49556e5b30a195af0e7aa298080e2b1873416ae6" exitCode=60 Nov 01 20:15:07.611932 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:15:07.611480 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-29b57c-29892735-2sz2s" event={"ID":"beccc923-162c-4600-a5b1-f6735a009583","Type":"ContainerDied","Data":"b12a5781c0d34b707edcc40c49556e5b30a195af0e7aa298080e2b1873416ae6"} Nov 01 20:15:07.611932 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:15:07.611598 5935 scope.go:122] "RemoveContainer" containerID="f6f048593531350d31227f55b95391a5bdafdfeda28f2b9a31f6cdc9eb04bacb" Nov 01 20:15:07.611932 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:15:07.611696 5935 scope.go:122] "RemoveContainer" containerID="b12a5781c0d34b707edcc40c49556e5b30a195af0e7aa298080e2b1873416ae6" Nov 01 20:15:07.612075 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:15:07.611956 5935 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-29b57c-29892735-2sz2s_odh-ai-gateway-infra(beccc923-162c-4600-a5b1-f6735a009583)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-29b57c-29892735-2sz2s" podUID="beccc923-162c-4600-a5b1-f6735a009583" Nov 01 20:15:08.619451 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:15:08.619425 5935 scope.go:122] "RemoveContainer" containerID="b12a5781c0d34b707edcc40c49556e5b30a195af0e7aa298080e2b1873416ae6" Nov 01 20:15:08.619825 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:15:08.619674 5935 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-29b57c-29892735-2sz2s_odh-ai-gateway-infra(beccc923-162c-4600-a5b1-f6735a009583)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-29b57c-29892735-2sz2s" podUID="beccc923-162c-4600-a5b1-f6735a009583" Nov 01 20:15:17.346352 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:15:17.346321 5935 scope.go:122] "RemoveContainer" containerID="b12a5781c0d34b707edcc40c49556e5b30a195af0e7aa298080e2b1873416ae6" Nov 01 20:15:17.722678 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:15:17.722642 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-29b57c-29892735-2sz2s" event={"ID":"beccc923-162c-4600-a5b1-f6735a009583","Type":"ContainerStarted","Data":"637114a83c4fd4a9144963af44399f8edc74fc5cd5ae8d073ab86df93966fe58"} Nov 01 20:15:18.734665 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:15:18.734623 5935 generic.go:356] "Generic (PLEG): container finished" podID="beccc923-162c-4600-a5b1-f6735a009583" containerID="637114a83c4fd4a9144963af44399f8edc74fc5cd5ae8d073ab86df93966fe58" exitCode=60 Nov 01 20:15:18.735124 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:15:18.734691 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-29b57c-29892735-2sz2s" event={"ID":"beccc923-162c-4600-a5b1-f6735a009583","Type":"ContainerDied","Data":"637114a83c4fd4a9144963af44399f8edc74fc5cd5ae8d073ab86df93966fe58"} Nov 01 20:15:18.735124 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:15:18.734760 5935 scope.go:122] "RemoveContainer" containerID="b12a5781c0d34b707edcc40c49556e5b30a195af0e7aa298080e2b1873416ae6" Nov 01 20:15:18.735214 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:15:18.735132 5935 scope.go:122] "RemoveContainer" containerID="637114a83c4fd4a9144963af44399f8edc74fc5cd5ae8d073ab86df93966fe58" Nov 01 20:15:18.735493 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:15:18.735470 5935 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-29b57c-29892735-2sz2s_odh-ai-gateway-infra(beccc923-162c-4600-a5b1-f6735a009583)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-29b57c-29892735-2sz2s" podUID="beccc923-162c-4600-a5b1-f6735a009583" Nov 01 20:15:18.774791 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:15:18.774765 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-29b57c-29892735-2sz2s"] Nov 01 20:15:19.909120 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:15:19.908892 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-29b57c-29892735-2sz2s" Nov 01 20:15:20.009345 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:15:20.009264 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/beccc923-162c-4600-a5b1-f6735a009583-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/beccc923-162c-4600-a5b1-f6735a009583-maas-api-service-ca\") pod \"beccc923-162c-4600-a5b1-f6735a009583\" (UID: \"beccc923-162c-4600-a5b1-f6735a009583\") " Nov 01 20:15:20.009345 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:15:20.009344 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/beccc923-162c-4600-a5b1-f6735a009583-kube-api-access-9cdh8\" (UniqueName: \"kubernetes.io/projected/beccc923-162c-4600-a5b1-f6735a009583-kube-api-access-9cdh8\") pod \"beccc923-162c-4600-a5b1-f6735a009583\" (UID: \"beccc923-162c-4600-a5b1-f6735a009583\") " Nov 01 20:15:20.009679 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:15:20.009650 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/beccc923-162c-4600-a5b1-f6735a009583-maas-api-service-ca" pod "beccc923-162c-4600-a5b1-f6735a009583" (UID: "beccc923-162c-4600-a5b1-f6735a009583"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:15:20.009734 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:15:20.009722 5935 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/beccc923-162c-4600-a5b1-f6735a009583-maas-api-service-ca\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:15:20.011403 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:15:20.011379 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beccc923-162c-4600-a5b1-f6735a009583-kube-api-access-9cdh8" pod "beccc923-162c-4600-a5b1-f6735a009583" (UID: "beccc923-162c-4600-a5b1-f6735a009583"). InnerVolumeSpecName "kube-api-access-9cdh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:15:20.110740 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:15:20.110710 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cdh8\" (UniqueName: \"kubernetes.io/projected/beccc923-162c-4600-a5b1-f6735a009583-kube-api-access-9cdh8\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:15:20.751032 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:15:20.751000 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-29b57c-29892735-2sz2s" Nov 01 20:15:20.751201 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:15:20.751037 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-29b57c-29892735-2sz2s" event={"ID":"beccc923-162c-4600-a5b1-f6735a009583","Type":"ContainerDied","Data":"a3e29fa233248fa046c9c2760e2a2a0af12f759e52774ed7a1cc6150eb7c2d3a"} Nov 01 20:15:20.751201 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:15:20.751101 5935 scope.go:122] "RemoveContainer" containerID="637114a83c4fd4a9144963af44399f8edc74fc5cd5ae8d073ab86df93966fe58" Nov 01 20:15:20.778459 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:15:20.778432 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-29b57c-29892735-2sz2s"] Nov 01 20:15:20.783130 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:15:20.783109 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-29b57c-29892735-2sz2s"] Nov 01 20:15:22.353248 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:15:22.353212 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="beccc923-162c-4600-a5b1-f6735a009583" path="/var/lib/kubelet/pods/beccc923-162c-4600-a5b1-f6735a009583/volumes" Nov 01 20:15:35.845550 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:15:35.845513 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:16:05.844110 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:16:05.844062 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:16:35.843658 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:16:35.843620 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:16:48.373627 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:48.373593 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x"] Nov 01 20:16:48.374175 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:48.374160 5935 cpu_manager.go:414] "containerMap: removing container" podUID="beccc923-162c-4600-a5b1-f6735a009583" containerName="cleanup" Nov 01 20:16:48.374219 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:48.374177 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="beccc923-162c-4600-a5b1-f6735a009583" containerName="cleanup" Nov 01 20:16:48.374219 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:48.374193 5935 cpu_manager.go:414] "containerMap: removing container" podUID="beccc923-162c-4600-a5b1-f6735a009583" containerName="cleanup" Nov 01 20:16:48.374219 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:48.374199 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="beccc923-162c-4600-a5b1-f6735a009583" containerName="cleanup" Nov 01 20:16:48.374219 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:48.374210 5935 cpu_manager.go:414] "containerMap: removing container" podUID="beccc923-162c-4600-a5b1-f6735a009583" containerName="cleanup" Nov 01 20:16:48.374219 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:48.374215 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="beccc923-162c-4600-a5b1-f6735a009583" containerName="cleanup" Nov 01 20:16:48.374413 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:48.374327 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="beccc923-162c-4600-a5b1-f6735a009583" containerName="cleanup" Nov 01 20:16:48.374413 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:48.374335 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="beccc923-162c-4600-a5b1-f6735a009583" containerName="cleanup" Nov 01 20:16:48.383030 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:48.383012 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x" Nov 01 20:16:48.387553 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:48.387527 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-29b57c-openshift-default-dockercfg-6fnqw\"" Nov 01 20:16:48.389692 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:48.389670 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x"] Nov 01 20:16:48.469977 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:48.469945 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b63dca7-1aeb-40c5-8b7c-328578a58adf-workload-certs\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x\" (UID: \"3b63dca7-1aeb-40c5-8b7c-328578a58adf\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x" Nov 01 20:16:48.469977 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:48.469977 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b63dca7-1aeb-40c5-8b7c-328578a58adf-istio-envoy\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x\" (UID: \"3b63dca7-1aeb-40c5-8b7c-328578a58adf\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x" Nov 01 20:16:48.470316 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:48.470140 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b63dca7-1aeb-40c5-8b7c-328578a58adf-istio-token\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x\" (UID: \"3b63dca7-1aeb-40c5-8b7c-328578a58adf\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x" Nov 01 20:16:48.470316 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:48.470198 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b63dca7-1aeb-40c5-8b7c-328578a58adf-istiod-ca-cert\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x\" (UID: \"3b63dca7-1aeb-40c5-8b7c-328578a58adf\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x" Nov 01 20:16:48.470445 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:48.470407 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b63dca7-1aeb-40c5-8b7c-328578a58adf-istio-podinfo\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x\" (UID: \"3b63dca7-1aeb-40c5-8b7c-328578a58adf\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x" Nov 01 20:16:48.470495 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:48.470452 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b63dca7-1aeb-40c5-8b7c-328578a58adf-workload-socket\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x\" (UID: \"3b63dca7-1aeb-40c5-8b7c-328578a58adf\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x" Nov 01 20:16:48.470663 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:48.470639 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b63dca7-1aeb-40c5-8b7c-328578a58adf-istio-data\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x\" (UID: \"3b63dca7-1aeb-40c5-8b7c-328578a58adf\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x" Nov 01 20:16:48.470725 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:48.470678 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46zxz\" (UniqueName: \"kubernetes.io/projected/3b63dca7-1aeb-40c5-8b7c-328578a58adf-kube-api-access-46zxz\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x\" (UID: \"3b63dca7-1aeb-40c5-8b7c-328578a58adf\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x" Nov 01 20:16:48.470725 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:48.470711 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b63dca7-1aeb-40c5-8b7c-328578a58adf-credential-socket\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x\" (UID: \"3b63dca7-1aeb-40c5-8b7c-328578a58adf\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x" Nov 01 20:16:48.572427 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:48.572393 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b63dca7-1aeb-40c5-8b7c-328578a58adf-istio-data\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x\" (UID: \"3b63dca7-1aeb-40c5-8b7c-328578a58adf\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x" Nov 01 20:16:48.572427 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:48.572430 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46zxz\" (UniqueName: \"kubernetes.io/projected/3b63dca7-1aeb-40c5-8b7c-328578a58adf-kube-api-access-46zxz\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x\" (UID: \"3b63dca7-1aeb-40c5-8b7c-328578a58adf\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x" Nov 01 20:16:48.572625 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:48.572457 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b63dca7-1aeb-40c5-8b7c-328578a58adf-credential-socket\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x\" (UID: \"3b63dca7-1aeb-40c5-8b7c-328578a58adf\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x" Nov 01 20:16:48.572625 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:48.572583 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b63dca7-1aeb-40c5-8b7c-328578a58adf-workload-certs\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x\" (UID: \"3b63dca7-1aeb-40c5-8b7c-328578a58adf\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x" Nov 01 20:16:48.572625 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:48.572607 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b63dca7-1aeb-40c5-8b7c-328578a58adf-istio-envoy\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x\" (UID: \"3b63dca7-1aeb-40c5-8b7c-328578a58adf\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x" Nov 01 20:16:48.572760 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:48.572669 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b63dca7-1aeb-40c5-8b7c-328578a58adf-istio-token\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x\" (UID: \"3b63dca7-1aeb-40c5-8b7c-328578a58adf\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x" Nov 01 20:16:48.572807 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:48.572768 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b63dca7-1aeb-40c5-8b7c-328578a58adf-istiod-ca-cert\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x\" (UID: \"3b63dca7-1aeb-40c5-8b7c-328578a58adf\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x" Nov 01 20:16:48.572853 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:48.572832 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b63dca7-1aeb-40c5-8b7c-328578a58adf-istio-podinfo\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x\" (UID: \"3b63dca7-1aeb-40c5-8b7c-328578a58adf\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x" Nov 01 20:16:48.572903 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:48.572864 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b63dca7-1aeb-40c5-8b7c-328578a58adf-workload-socket\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x\" (UID: \"3b63dca7-1aeb-40c5-8b7c-328578a58adf\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x" Nov 01 20:16:48.572958 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:48.572897 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b63dca7-1aeb-40c5-8b7c-328578a58adf-credential-socket\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x\" (UID: \"3b63dca7-1aeb-40c5-8b7c-328578a58adf\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x" Nov 01 20:16:48.572958 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:48.572942 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b63dca7-1aeb-40c5-8b7c-328578a58adf-istio-data\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x\" (UID: \"3b63dca7-1aeb-40c5-8b7c-328578a58adf\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x" Nov 01 20:16:48.573309 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:48.573242 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b63dca7-1aeb-40c5-8b7c-328578a58adf-workload-socket\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x\" (UID: \"3b63dca7-1aeb-40c5-8b7c-328578a58adf\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x" Nov 01 20:16:48.573513 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:48.573489 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b63dca7-1aeb-40c5-8b7c-328578a58adf-workload-certs\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x\" (UID: \"3b63dca7-1aeb-40c5-8b7c-328578a58adf\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x" Nov 01 20:16:48.573778 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:48.573751 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b63dca7-1aeb-40c5-8b7c-328578a58adf-istiod-ca-cert\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x\" (UID: \"3b63dca7-1aeb-40c5-8b7c-328578a58adf\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x" Nov 01 20:16:48.576760 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:48.576736 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b63dca7-1aeb-40c5-8b7c-328578a58adf-istio-podinfo\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x\" (UID: \"3b63dca7-1aeb-40c5-8b7c-328578a58adf\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x" Nov 01 20:16:48.576926 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:48.576757 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b63dca7-1aeb-40c5-8b7c-328578a58adf-istio-envoy\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x\" (UID: \"3b63dca7-1aeb-40c5-8b7c-328578a58adf\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x" Nov 01 20:16:48.580677 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:48.580649 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b63dca7-1aeb-40c5-8b7c-328578a58adf-istio-token\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x\" (UID: \"3b63dca7-1aeb-40c5-8b7c-328578a58adf\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x" Nov 01 20:16:48.581872 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:48.581847 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46zxz\" (UniqueName: \"kubernetes.io/projected/3b63dca7-1aeb-40c5-8b7c-328578a58adf-kube-api-access-46zxz\") pod \"e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x\" (UID: \"3b63dca7-1aeb-40c5-8b7c-328578a58adf\") " pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x" Nov 01 20:16:48.700166 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:48.700140 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x" Nov 01 20:16:48.984013 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:48.983980 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x"] Nov 01 20:16:48.985811 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:16:48.985772 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b63dca7_1aeb_40c5_8b7c_328578a58adf.slice/crio-209b305956d735e360d90ba27ee6dee124cbb7ae99cedf4b3ea1e6312da29e36 WatchSource:0}: Error finding container 209b305956d735e360d90ba27ee6dee124cbb7ae99cedf4b3ea1e6312da29e36: Status 404 returned error can't find the container with id 209b305956d735e360d90ba27ee6dee124cbb7ae99cedf4b3ea1e6312da29e36 Nov 01 20:16:48.992279 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:48.992246 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:16:48.992400 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:48.992344 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:16:48.992400 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:48.992380 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:16:49.453312 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:49.451181 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x" event={"ID":"3b63dca7-1aeb-40c5-8b7c-328578a58adf","Type":"ContainerStarted","Data":"caf0c3b20c44c875936c8eff8a6a59738c096cad97f7651a9bee823ebb5d9bb5"} Nov 01 20:16:49.453312 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:49.451238 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x" event={"ID":"3b63dca7-1aeb-40c5-8b7c-328578a58adf","Type":"ContainerStarted","Data":"209b305956d735e360d90ba27ee6dee124cbb7ae99cedf4b3ea1e6312da29e36"} Nov 01 20:16:49.481172 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:49.481117 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x" podStartSLOduration=1.481100458 podStartE2EDuration="1.481100458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:16:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:16:49.478724062 +0000 UTC m=+1055.251884575" watchObservedRunningTime="2026-11-01 20:16:49.481100458 +0000 UTC m=+1055.254260967" Nov 01 20:16:49.700613 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:49.700577 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x" Nov 01 20:16:50.702458 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:50.702425 5935 patch_prober.go:29] interesting pod/e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 01 20:16:50.702858 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:50.702474 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x" podUID="3b63dca7-1aeb-40c5-8b7c-328578a58adf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 01 20:16:51.701600 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:51.701566 5935 patch_prober.go:29] interesting pod/e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 01 20:16:51.701772 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:51.701625 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x" podUID="3b63dca7-1aeb-40c5-8b7c-328578a58adf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 01 20:16:52.701385 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:52.701355 5935 patch_prober.go:29] interesting pod/e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 01 20:16:52.702300 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:52.701401 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x" podUID="3b63dca7-1aeb-40c5-8b7c-328578a58adf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 01 20:16:53.700580 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:53.700548 5935 patch_prober.go:29] interesting pod/e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 01 20:16:53.700838 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:53.700599 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x" podUID="3b63dca7-1aeb-40c5-8b7c-328578a58adf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 01 20:16:54.700719 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:54.700683 5935 patch_prober.go:29] interesting pod/e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Nov 01 20:16:54.701108 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:54.700739 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x" podUID="3b63dca7-1aeb-40c5-8b7c-328578a58adf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Nov 01 20:16:55.704662 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:55.704633 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x" Nov 01 20:16:55.705093 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:55.705071 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x" Nov 01 20:16:55.705859 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:55.705839 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x" Nov 01 20:16:59.185332 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:59.185297 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e83b4b-86c6f68dcf-zbtwx"] Nov 01 20:16:59.186326 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:59.186307 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="beccc923-162c-4600-a5b1-f6735a009583" containerName="cleanup" Nov 01 20:16:59.189916 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:59.189898 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e83b4b-86c6f68dcf-zbtwx" Nov 01 20:16:59.198615 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:59.198578 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-e83b4b-serving-cert\"" Nov 01 20:16:59.202744 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:59.202696 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e83b4b-86c6f68dcf-zbtwx"] Nov 01 20:16:59.359480 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:59.359441 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2pml\" (UniqueName: \"kubernetes.io/projected/acbd9a98-25e3-4dd9-9f27-2cf274a72c02-kube-api-access-b2pml\") pod \"maas-api-e2e-shared-b-w1-e83b4b-86c6f68dcf-zbtwx\" (UID: \"acbd9a98-25e3-4dd9-9f27-2cf274a72c02\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e83b4b-86c6f68dcf-zbtwx" Nov 01 20:16:59.359661 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:59.359487 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acbd9a98-25e3-4dd9-9f27-2cf274a72c02-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-e83b4b-86c6f68dcf-zbtwx\" (UID: \"acbd9a98-25e3-4dd9-9f27-2cf274a72c02\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e83b4b-86c6f68dcf-zbtwx" Nov 01 20:16:59.359661 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:59.359614 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acbd9a98-25e3-4dd9-9f27-2cf274a72c02-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-e83b4b-86c6f68dcf-zbtwx\" (UID: \"acbd9a98-25e3-4dd9-9f27-2cf274a72c02\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e83b4b-86c6f68dcf-zbtwx" Nov 01 20:16:59.359661 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:59.359658 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acbd9a98-25e3-4dd9-9f27-2cf274a72c02-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-e83b4b-86c6f68dcf-zbtwx\" (UID: \"acbd9a98-25e3-4dd9-9f27-2cf274a72c02\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e83b4b-86c6f68dcf-zbtwx" Nov 01 20:16:59.461426 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:59.461385 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2pml\" (UniqueName: \"kubernetes.io/projected/acbd9a98-25e3-4dd9-9f27-2cf274a72c02-kube-api-access-b2pml\") pod \"maas-api-e2e-shared-b-w1-e83b4b-86c6f68dcf-zbtwx\" (UID: \"acbd9a98-25e3-4dd9-9f27-2cf274a72c02\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e83b4b-86c6f68dcf-zbtwx" Nov 01 20:16:59.461632 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:59.461442 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acbd9a98-25e3-4dd9-9f27-2cf274a72c02-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-e83b4b-86c6f68dcf-zbtwx\" (UID: \"acbd9a98-25e3-4dd9-9f27-2cf274a72c02\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e83b4b-86c6f68dcf-zbtwx" Nov 01 20:16:59.461632 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:59.461523 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acbd9a98-25e3-4dd9-9f27-2cf274a72c02-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-e83b4b-86c6f68dcf-zbtwx\" (UID: \"acbd9a98-25e3-4dd9-9f27-2cf274a72c02\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e83b4b-86c6f68dcf-zbtwx" Nov 01 20:16:59.461920 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:59.461893 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acbd9a98-25e3-4dd9-9f27-2cf274a72c02-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-e83b4b-86c6f68dcf-zbtwx\" (UID: \"acbd9a98-25e3-4dd9-9f27-2cf274a72c02\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e83b4b-86c6f68dcf-zbtwx" Nov 01 20:16:59.462720 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:59.462692 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acbd9a98-25e3-4dd9-9f27-2cf274a72c02-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-e83b4b-86c6f68dcf-zbtwx\" (UID: \"acbd9a98-25e3-4dd9-9f27-2cf274a72c02\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e83b4b-86c6f68dcf-zbtwx" Nov 01 20:16:59.464389 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:59.464359 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acbd9a98-25e3-4dd9-9f27-2cf274a72c02-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-e83b4b-86c6f68dcf-zbtwx\" (UID: \"acbd9a98-25e3-4dd9-9f27-2cf274a72c02\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e83b4b-86c6f68dcf-zbtwx" Nov 01 20:16:59.464517 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:59.464500 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acbd9a98-25e3-4dd9-9f27-2cf274a72c02-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-e83b4b-86c6f68dcf-zbtwx\" (UID: \"acbd9a98-25e3-4dd9-9f27-2cf274a72c02\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e83b4b-86c6f68dcf-zbtwx" Nov 01 20:16:59.471079 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:59.471053 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2pml\" (UniqueName: \"kubernetes.io/projected/acbd9a98-25e3-4dd9-9f27-2cf274a72c02-kube-api-access-b2pml\") pod \"maas-api-e2e-shared-b-w1-e83b4b-86c6f68dcf-zbtwx\" (UID: \"acbd9a98-25e3-4dd9-9f27-2cf274a72c02\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e83b4b-86c6f68dcf-zbtwx" Nov 01 20:16:59.515648 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:59.515610 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e83b4b-86c6f68dcf-zbtwx" Nov 01 20:16:59.810992 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:16:59.810966 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e83b4b-86c6f68dcf-zbtwx"] Nov 01 20:16:59.812051 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:16:59.812027 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacbd9a98_25e3_4dd9_9f27_2cf274a72c02.slice/crio-27e021b97fe1a3e9f1c96cdfee54a9928830dccfcb3d5dced1d674a49b0240f4 WatchSource:0}: Error finding container 27e021b97fe1a3e9f1c96cdfee54a9928830dccfcb3d5dced1d674a49b0240f4: Status 404 returned error can't find the container with id 27e021b97fe1a3e9f1c96cdfee54a9928830dccfcb3d5dced1d674a49b0240f4 Nov 01 20:17:00.560412 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:00.560330 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e83b4b-86c6f68dcf-zbtwx" event={"ID":"acbd9a98-25e3-4dd9-9f27-2cf274a72c02","Type":"ContainerStarted","Data":"ed542080fd41daf8476fe0862169697337b18c520ba465d5fb12f7ab2ec80a08"} Nov 01 20:17:00.560412 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:00.560404 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e83b4b-86c6f68dcf-zbtwx" event={"ID":"acbd9a98-25e3-4dd9-9f27-2cf274a72c02","Type":"ContainerStarted","Data":"27e021b97fe1a3e9f1c96cdfee54a9928830dccfcb3d5dced1d674a49b0240f4"} Nov 01 20:17:00.560860 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:00.560467 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e83b4b-86c6f68dcf-zbtwx" Nov 01 20:17:00.578102 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:00.578058 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e83b4b-86c6f68dcf-zbtwx" podStartSLOduration=1.166476042 podStartE2EDuration="1.57804501s" totalImagesPullingTime="411.568968ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:16:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:17:00.576210645 +0000 UTC m=+1066.349371158" watchObservedRunningTime="2026-11-01 20:17:00.57804501 +0000 UTC m=+1066.351205551" Nov 01 20:17:05.846983 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:17:05.846935 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:17:06.572670 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:06.572641 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e83b4b-86c6f68dcf-zbtwx" Nov 01 20:17:16.917329 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:16.917003 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-9be196/auth-test-model-9be196-kserve-587468665c-b2xdw"] Nov 01 20:17:16.922241 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:16.922213 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-9be196/auth-test-model-9be196-kserve-587468665c-b2xdw" Nov 01 20:17:16.930482 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:16.930459 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-9be196\"/\"openshift-service-ca.crt\"" Nov 01 20:17:16.931174 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:16.930926 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-9be196\"/\"kube-root-ca.crt\"" Nov 01 20:17:16.931330 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:16.931040 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-9be196\"/\"default-dockercfg-fzplz\"" Nov 01 20:17:16.934801 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:16.933036 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-9be196/auth-test-model-9be196-kserve-587468665c-b2xdw"] Nov 01 20:17:16.944436 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:16.944411 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-9be196\"/\"auth-test-model-9be196-kserve-self-signed-certs\"" Nov 01 20:17:16.980788 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:16.980762 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffc910ff-b218-4ea3-893e-987a9503a634-home\") pod \"auth-test-model-9be196-kserve-587468665c-b2xdw\" (UID: \"ffc910ff-b218-4ea3-893e-987a9503a634\") " pod="ai-tenant-e2e-shared-a-w1-9be196/auth-test-model-9be196-kserve-587468665c-b2xdw" Nov 01 20:17:16.980788 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:16.980791 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffc910ff-b218-4ea3-893e-987a9503a634-tls-certs\") pod \"auth-test-model-9be196-kserve-587468665c-b2xdw\" (UID: \"ffc910ff-b218-4ea3-893e-987a9503a634\") " pod="ai-tenant-e2e-shared-a-w1-9be196/auth-test-model-9be196-kserve-587468665c-b2xdw" Nov 01 20:17:16.980996 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:16.980844 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffc910ff-b218-4ea3-893e-987a9503a634-dshm\") pod \"auth-test-model-9be196-kserve-587468665c-b2xdw\" (UID: \"ffc910ff-b218-4ea3-893e-987a9503a634\") " pod="ai-tenant-e2e-shared-a-w1-9be196/auth-test-model-9be196-kserve-587468665c-b2xdw" Nov 01 20:17:16.980996 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:16.980945 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffc910ff-b218-4ea3-893e-987a9503a634-model-cache\") pod \"auth-test-model-9be196-kserve-587468665c-b2xdw\" (UID: \"ffc910ff-b218-4ea3-893e-987a9503a634\") " pod="ai-tenant-e2e-shared-a-w1-9be196/auth-test-model-9be196-kserve-587468665c-b2xdw" Nov 01 20:17:16.980996 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:16.980975 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffc910ff-b218-4ea3-893e-987a9503a634-tmp-dir\") pod \"auth-test-model-9be196-kserve-587468665c-b2xdw\" (UID: \"ffc910ff-b218-4ea3-893e-987a9503a634\") " pod="ai-tenant-e2e-shared-a-w1-9be196/auth-test-model-9be196-kserve-587468665c-b2xdw" Nov 01 20:17:16.980996 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:16.980993 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z667l\" (UniqueName: \"kubernetes.io/projected/ffc910ff-b218-4ea3-893e-987a9503a634-kube-api-access-z667l\") pod \"auth-test-model-9be196-kserve-587468665c-b2xdw\" (UID: \"ffc910ff-b218-4ea3-893e-987a9503a634\") " pod="ai-tenant-e2e-shared-a-w1-9be196/auth-test-model-9be196-kserve-587468665c-b2xdw" Nov 01 20:17:17.083019 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:17.082965 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffc910ff-b218-4ea3-893e-987a9503a634-home\") pod \"auth-test-model-9be196-kserve-587468665c-b2xdw\" (UID: \"ffc910ff-b218-4ea3-893e-987a9503a634\") " pod="ai-tenant-e2e-shared-a-w1-9be196/auth-test-model-9be196-kserve-587468665c-b2xdw" Nov 01 20:17:17.083019 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:17.083017 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffc910ff-b218-4ea3-893e-987a9503a634-tls-certs\") pod \"auth-test-model-9be196-kserve-587468665c-b2xdw\" (UID: \"ffc910ff-b218-4ea3-893e-987a9503a634\") " pod="ai-tenant-e2e-shared-a-w1-9be196/auth-test-model-9be196-kserve-587468665c-b2xdw" Nov 01 20:17:17.083271 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:17.083106 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffc910ff-b218-4ea3-893e-987a9503a634-dshm\") pod \"auth-test-model-9be196-kserve-587468665c-b2xdw\" (UID: \"ffc910ff-b218-4ea3-893e-987a9503a634\") " pod="ai-tenant-e2e-shared-a-w1-9be196/auth-test-model-9be196-kserve-587468665c-b2xdw" Nov 01 20:17:17.083271 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:17.083177 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffc910ff-b218-4ea3-893e-987a9503a634-model-cache\") pod \"auth-test-model-9be196-kserve-587468665c-b2xdw\" (UID: \"ffc910ff-b218-4ea3-893e-987a9503a634\") " pod="ai-tenant-e2e-shared-a-w1-9be196/auth-test-model-9be196-kserve-587468665c-b2xdw" Nov 01 20:17:17.083271 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:17.083208 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffc910ff-b218-4ea3-893e-987a9503a634-tmp-dir\") pod \"auth-test-model-9be196-kserve-587468665c-b2xdw\" (UID: \"ffc910ff-b218-4ea3-893e-987a9503a634\") " pod="ai-tenant-e2e-shared-a-w1-9be196/auth-test-model-9be196-kserve-587468665c-b2xdw" Nov 01 20:17:17.083271 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:17.083233 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z667l\" (UniqueName: \"kubernetes.io/projected/ffc910ff-b218-4ea3-893e-987a9503a634-kube-api-access-z667l\") pod \"auth-test-model-9be196-kserve-587468665c-b2xdw\" (UID: \"ffc910ff-b218-4ea3-893e-987a9503a634\") " pod="ai-tenant-e2e-shared-a-w1-9be196/auth-test-model-9be196-kserve-587468665c-b2xdw" Nov 01 20:17:17.084120 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:17.084094 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffc910ff-b218-4ea3-893e-987a9503a634-model-cache\") pod \"auth-test-model-9be196-kserve-587468665c-b2xdw\" (UID: \"ffc910ff-b218-4ea3-893e-987a9503a634\") " pod="ai-tenant-e2e-shared-a-w1-9be196/auth-test-model-9be196-kserve-587468665c-b2xdw" Nov 01 20:17:17.084485 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:17.084463 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffc910ff-b218-4ea3-893e-987a9503a634-tmp-dir\") pod \"auth-test-model-9be196-kserve-587468665c-b2xdw\" (UID: \"ffc910ff-b218-4ea3-893e-987a9503a634\") " pod="ai-tenant-e2e-shared-a-w1-9be196/auth-test-model-9be196-kserve-587468665c-b2xdw" Nov 01 20:17:17.084683 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:17.084538 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffc910ff-b218-4ea3-893e-987a9503a634-home\") pod \"auth-test-model-9be196-kserve-587468665c-b2xdw\" (UID: \"ffc910ff-b218-4ea3-893e-987a9503a634\") " pod="ai-tenant-e2e-shared-a-w1-9be196/auth-test-model-9be196-kserve-587468665c-b2xdw" Nov 01 20:17:17.086211 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:17.086185 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffc910ff-b218-4ea3-893e-987a9503a634-tls-certs\") pod \"auth-test-model-9be196-kserve-587468665c-b2xdw\" (UID: \"ffc910ff-b218-4ea3-893e-987a9503a634\") " pod="ai-tenant-e2e-shared-a-w1-9be196/auth-test-model-9be196-kserve-587468665c-b2xdw" Nov 01 20:17:17.104802 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:17.104537 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffc910ff-b218-4ea3-893e-987a9503a634-dshm\") pod \"auth-test-model-9be196-kserve-587468665c-b2xdw\" (UID: \"ffc910ff-b218-4ea3-893e-987a9503a634\") " pod="ai-tenant-e2e-shared-a-w1-9be196/auth-test-model-9be196-kserve-587468665c-b2xdw" Nov 01 20:17:17.111310 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:17.109405 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z667l\" (UniqueName: \"kubernetes.io/projected/ffc910ff-b218-4ea3-893e-987a9503a634-kube-api-access-z667l\") pod \"auth-test-model-9be196-kserve-587468665c-b2xdw\" (UID: \"ffc910ff-b218-4ea3-893e-987a9503a634\") " pod="ai-tenant-e2e-shared-a-w1-9be196/auth-test-model-9be196-kserve-587468665c-b2xdw" Nov 01 20:17:17.264786 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:17.264751 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-9be196/auth-test-model-9be196-kserve-587468665c-b2xdw" Nov 01 20:17:17.558096 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:17.558063 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-9be196/auth-test-model-9be196-kserve-587468665c-b2xdw"] Nov 01 20:17:17.559068 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:17:17.559037 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podffc910ff_b218_4ea3_893e_987a9503a634.slice/crio-2be413a6355d00bd3a697c4daae6cbccc5290cc1ee06def6de227934ec6f1a5d WatchSource:0}: Error finding container 2be413a6355d00bd3a697c4daae6cbccc5290cc1ee06def6de227934ec6f1a5d: Status 404 returned error can't find the container with id 2be413a6355d00bd3a697c4daae6cbccc5290cc1ee06def6de227934ec6f1a5d Nov 01 20:17:17.682094 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:17.682062 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-9be196/auth-test-model-9be196-kserve-587468665c-b2xdw" event={"ID":"ffc910ff-b218-4ea3-893e-987a9503a634","Type":"ContainerStarted","Data":"790244db417be08a390c18e63999b627438522bacd86bf36805083fb1031460e"} Nov 01 20:17:17.682249 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:17.682108 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-9be196/auth-test-model-9be196-kserve-587468665c-b2xdw" event={"ID":"ffc910ff-b218-4ea3-893e-987a9503a634","Type":"ContainerStarted","Data":"2be413a6355d00bd3a697c4daae6cbccc5290cc1ee06def6de227934ec6f1a5d"} Nov 01 20:17:17.699511 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:17.699466 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-9be196/auth-test-model-9be196-kserve-587468665c-b2xdw" podStartSLOduration=1.699453912 podStartE2EDuration="1.699453912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:17:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:17:17.696442267 +0000 UTC m=+1083.469602779" watchObservedRunningTime="2026-11-01 20:17:17.699453912 +0000 UTC m=+1083.472614459" Nov 01 20:17:18.265752 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:18.265721 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-9be196/auth-test-model-9be196-kserve-587468665c-b2xdw" Nov 01 20:17:27.265208 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:27.265110 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-9be196/auth-test-model-9be196-kserve-587468665c-b2xdw" Nov 01 20:17:27.278038 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:27.278017 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-9be196/auth-test-model-9be196-kserve-587468665c-b2xdw" Nov 01 20:17:27.772065 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:27.772034 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-9be196/auth-test-model-9be196-kserve-587468665c-b2xdw" Nov 01 20:17:28.487275 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:28.487243 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-849d6msts"] Nov 01 20:17:28.491609 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:28.491590 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-849d6msts" Nov 01 20:17:28.496678 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:28.496652 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-aaea41\"/\"routing-overlay\"" Nov 01 20:17:28.499042 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:28.498788 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-aaea41\"/\"kube-root-ca.crt\"" Nov 01 20:17:28.499042 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:28.498834 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-aaea41\"/\"openshift-service-ca.crt\"" Nov 01 20:17:28.499042 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:28.498848 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-aaea41\"/\"payload-processing-external-model-plugins-e2e-extmod-a-aaea41\"" Nov 01 20:17:28.499042 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:28.498847 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-aaea41\"/\"payload-processing-external-model-e2e--245da986-dockercfg-r9b8m\"" Nov 01 20:17:28.499042 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:28.498878 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-aaea41\"/\"agc-cred-b2b285556e64\"" Nov 01 20:17:28.499042 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:28.498885 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-aaea41\"/\"e2e-cfl-e2318b-provider-cred\"" Nov 01 20:17:28.503079 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:28.503051 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-849d6msts"] Nov 01 20:17:28.642707 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:28.642674 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bb8ccaa1-ea3f-4f91-ad5d-173a28d77459-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-849d6msts\" (UID: \"bb8ccaa1-ea3f-4f91-ad5d-173a28d77459\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-849d6msts" Nov 01 20:17:28.642876 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:28.642783 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb8ccaa1-ea3f-4f91-ad5d-173a28d77459-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-849d6msts\" (UID: \"bb8ccaa1-ea3f-4f91-ad5d-173a28d77459\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-849d6msts" Nov 01 20:17:28.642876 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:28.642803 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bb8ccaa1-ea3f-4f91-ad5d-173a28d77459-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-849d6msts\" (UID: \"bb8ccaa1-ea3f-4f91-ad5d-173a28d77459\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-849d6msts" Nov 01 20:17:28.642949 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:28.642928 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpkch\" (UniqueName: \"kubernetes.io/projected/bb8ccaa1-ea3f-4f91-ad5d-173a28d77459-kube-api-access-kpkch\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-849d6msts\" (UID: \"bb8ccaa1-ea3f-4f91-ad5d-173a28d77459\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-849d6msts" Nov 01 20:17:28.744296 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:28.744186 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bb8ccaa1-ea3f-4f91-ad5d-173a28d77459-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-849d6msts\" (UID: \"bb8ccaa1-ea3f-4f91-ad5d-173a28d77459\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-849d6msts" Nov 01 20:17:28.744479 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:28.744329 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb8ccaa1-ea3f-4f91-ad5d-173a28d77459-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-849d6msts\" (UID: \"bb8ccaa1-ea3f-4f91-ad5d-173a28d77459\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-849d6msts" Nov 01 20:17:28.744479 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:28.744360 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bb8ccaa1-ea3f-4f91-ad5d-173a28d77459-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-849d6msts\" (UID: \"bb8ccaa1-ea3f-4f91-ad5d-173a28d77459\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-849d6msts" Nov 01 20:17:28.744479 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:28.744449 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpkch\" (UniqueName: \"kubernetes.io/projected/bb8ccaa1-ea3f-4f91-ad5d-173a28d77459-kube-api-access-kpkch\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-849d6msts\" (UID: \"bb8ccaa1-ea3f-4f91-ad5d-173a28d77459\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-849d6msts" Nov 01 20:17:28.745793 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:28.745763 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb8ccaa1-ea3f-4f91-ad5d-173a28d77459-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-849d6msts\" (UID: \"bb8ccaa1-ea3f-4f91-ad5d-173a28d77459\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-849d6msts" Nov 01 20:17:28.746018 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:17:28.746002 5935 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-aaea41/agc-cred-b2b285556e64: secret "agc-cred-b2b285556e64" not found Nov 01 20:17:28.746085 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:17:28.746027 5935 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-849d6msts: secret "agc-cred-b2b285556e64" not found Nov 01 20:17:28.746145 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:17:28.746114 5935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bb8ccaa1-ea3f-4f91-ad5d-173a28d77459-provider-credentials podName:bb8ccaa1-ea3f-4f91-ad5d-173a28d77459 nodeName:}" failed. No retries permitted until 2026-11-01 20:17:29.246091992 +0000 UTC m=+1095.019252503 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/bb8ccaa1-ea3f-4f91-ad5d-173a28d77459-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-aaea41-849d6msts" (UID: "bb8ccaa1-ea3f-4f91-ad5d-173a28d77459") : secret "agc-cred-b2b285556e64" not found Nov 01 20:17:28.747872 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:28.747842 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bb8ccaa1-ea3f-4f91-ad5d-173a28d77459-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-849d6msts\" (UID: \"bb8ccaa1-ea3f-4f91-ad5d-173a28d77459\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-849d6msts" Nov 01 20:17:28.761690 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:28.760785 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpkch\" (UniqueName: \"kubernetes.io/projected/bb8ccaa1-ea3f-4f91-ad5d-173a28d77459-kube-api-access-kpkch\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-849d6msts\" (UID: \"bb8ccaa1-ea3f-4f91-ad5d-173a28d77459\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-849d6msts" Nov 01 20:17:29.254867 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:29.254830 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bb8ccaa1-ea3f-4f91-ad5d-173a28d77459-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-849d6msts\" (UID: \"bb8ccaa1-ea3f-4f91-ad5d-173a28d77459\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-849d6msts" Nov 01 20:17:29.255044 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:17:29.254975 5935 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-aaea41/agc-cred-b2b285556e64: secret "agc-cred-b2b285556e64" not found Nov 01 20:17:29.255044 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:17:29.254999 5935 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-849d6msts: secret "agc-cred-b2b285556e64" not found Nov 01 20:17:29.255126 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:17:29.255056 5935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bb8ccaa1-ea3f-4f91-ad5d-173a28d77459-provider-credentials podName:bb8ccaa1-ea3f-4f91-ad5d-173a28d77459 nodeName:}" failed. No retries permitted until 2026-11-01 20:17:30.255039751 +0000 UTC m=+1096.028200259 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/bb8ccaa1-ea3f-4f91-ad5d-173a28d77459-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-aaea41-849d6msts" (UID: "bb8ccaa1-ea3f-4f91-ad5d-173a28d77459") : secret "agc-cred-b2b285556e64" not found Nov 01 20:17:29.769185 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:29.769159 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr"] Nov 01 20:17:29.773581 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:29.773559 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr" Nov 01 20:17:29.779462 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:29.779437 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-9de2ac-openshift-default-dockercfg-4kqbj\"" Nov 01 20:17:29.784349 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:29.784303 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr"] Nov 01 20:17:29.863335 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:29.863273 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4659fd62-58a7-4792-ad28-93f3c52c7d3a-credential-socket\") pod \"e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr\" (UID: \"4659fd62-58a7-4792-ad28-93f3c52c7d3a\") " pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr" Nov 01 20:17:29.863499 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:29.863386 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4659fd62-58a7-4792-ad28-93f3c52c7d3a-istio-podinfo\") pod \"e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr\" (UID: \"4659fd62-58a7-4792-ad28-93f3c52c7d3a\") " pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr" Nov 01 20:17:29.863499 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:29.863425 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4659fd62-58a7-4792-ad28-93f3c52c7d3a-workload-socket\") pod \"e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr\" (UID: \"4659fd62-58a7-4792-ad28-93f3c52c7d3a\") " pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr" Nov 01 20:17:29.863574 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:29.863514 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4659fd62-58a7-4792-ad28-93f3c52c7d3a-istio-data\") pod \"e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr\" (UID: \"4659fd62-58a7-4792-ad28-93f3c52c7d3a\") " pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr" Nov 01 20:17:29.863628 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:29.863616 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4659fd62-58a7-4792-ad28-93f3c52c7d3a-istio-envoy\") pod \"e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr\" (UID: \"4659fd62-58a7-4792-ad28-93f3c52c7d3a\") " pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr" Nov 01 20:17:29.863670 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:29.863656 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4659fd62-58a7-4792-ad28-93f3c52c7d3a-istiod-ca-cert\") pod \"e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr\" (UID: \"4659fd62-58a7-4792-ad28-93f3c52c7d3a\") " pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr" Nov 01 20:17:29.863707 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:29.863695 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4659fd62-58a7-4792-ad28-93f3c52c7d3a-istio-token\") pod \"e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr\" (UID: \"4659fd62-58a7-4792-ad28-93f3c52c7d3a\") " pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr" Nov 01 20:17:29.863793 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:29.863781 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pv55\" (UniqueName: \"kubernetes.io/projected/4659fd62-58a7-4792-ad28-93f3c52c7d3a-kube-api-access-4pv55\") pod \"e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr\" (UID: \"4659fd62-58a7-4792-ad28-93f3c52c7d3a\") " pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr" Nov 01 20:17:29.863828 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:29.863818 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4659fd62-58a7-4792-ad28-93f3c52c7d3a-workload-certs\") pod \"e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr\" (UID: \"4659fd62-58a7-4792-ad28-93f3c52c7d3a\") " pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr" Nov 01 20:17:29.964860 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:29.964824 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4659fd62-58a7-4792-ad28-93f3c52c7d3a-credential-socket\") pod \"e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr\" (UID: \"4659fd62-58a7-4792-ad28-93f3c52c7d3a\") " pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr" Nov 01 20:17:29.965043 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:29.964889 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4659fd62-58a7-4792-ad28-93f3c52c7d3a-istio-podinfo\") pod \"e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr\" (UID: \"4659fd62-58a7-4792-ad28-93f3c52c7d3a\") " pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr" Nov 01 20:17:29.965043 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:29.964917 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4659fd62-58a7-4792-ad28-93f3c52c7d3a-workload-socket\") pod \"e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr\" (UID: \"4659fd62-58a7-4792-ad28-93f3c52c7d3a\") " pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr" Nov 01 20:17:29.965043 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:29.964935 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4659fd62-58a7-4792-ad28-93f3c52c7d3a-istio-data\") pod \"e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr\" (UID: \"4659fd62-58a7-4792-ad28-93f3c52c7d3a\") " pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr" Nov 01 20:17:29.965206 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:29.965059 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4659fd62-58a7-4792-ad28-93f3c52c7d3a-istio-envoy\") pod \"e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr\" (UID: \"4659fd62-58a7-4792-ad28-93f3c52c7d3a\") " pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr" Nov 01 20:17:29.965206 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:29.965085 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4659fd62-58a7-4792-ad28-93f3c52c7d3a-istiod-ca-cert\") pod \"e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr\" (UID: \"4659fd62-58a7-4792-ad28-93f3c52c7d3a\") " pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr" Nov 01 20:17:29.965206 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:29.965140 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4659fd62-58a7-4792-ad28-93f3c52c7d3a-istio-token\") pod \"e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr\" (UID: \"4659fd62-58a7-4792-ad28-93f3c52c7d3a\") " pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr" Nov 01 20:17:29.965409 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:29.965223 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pv55\" (UniqueName: \"kubernetes.io/projected/4659fd62-58a7-4792-ad28-93f3c52c7d3a-kube-api-access-4pv55\") pod \"e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr\" (UID: \"4659fd62-58a7-4792-ad28-93f3c52c7d3a\") " pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr" Nov 01 20:17:29.965409 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:29.965227 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4659fd62-58a7-4792-ad28-93f3c52c7d3a-credential-socket\") pod \"e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr\" (UID: \"4659fd62-58a7-4792-ad28-93f3c52c7d3a\") " pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr" Nov 01 20:17:29.965409 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:29.965336 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4659fd62-58a7-4792-ad28-93f3c52c7d3a-workload-certs\") pod \"e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr\" (UID: \"4659fd62-58a7-4792-ad28-93f3c52c7d3a\") " pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr" Nov 01 20:17:29.965409 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:29.965386 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4659fd62-58a7-4792-ad28-93f3c52c7d3a-istio-data\") pod \"e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr\" (UID: \"4659fd62-58a7-4792-ad28-93f3c52c7d3a\") " pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr" Nov 01 20:17:29.965763 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:29.965740 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4659fd62-58a7-4792-ad28-93f3c52c7d3a-workload-certs\") pod \"e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr\" (UID: \"4659fd62-58a7-4792-ad28-93f3c52c7d3a\") " pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr" Nov 01 20:17:29.965857 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:29.965845 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4659fd62-58a7-4792-ad28-93f3c52c7d3a-istiod-ca-cert\") pod \"e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr\" (UID: \"4659fd62-58a7-4792-ad28-93f3c52c7d3a\") " pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr" Nov 01 20:17:29.965944 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:29.965925 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4659fd62-58a7-4792-ad28-93f3c52c7d3a-workload-socket\") pod \"e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr\" (UID: \"4659fd62-58a7-4792-ad28-93f3c52c7d3a\") " pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr" Nov 01 20:17:29.967417 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:29.967401 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4659fd62-58a7-4792-ad28-93f3c52c7d3a-istio-envoy\") pod \"e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr\" (UID: \"4659fd62-58a7-4792-ad28-93f3c52c7d3a\") " pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr" Nov 01 20:17:29.967482 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:29.967466 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4659fd62-58a7-4792-ad28-93f3c52c7d3a-istio-podinfo\") pod \"e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr\" (UID: \"4659fd62-58a7-4792-ad28-93f3c52c7d3a\") " pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr" Nov 01 20:17:29.972460 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:29.972438 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4659fd62-58a7-4792-ad28-93f3c52c7d3a-istio-token\") pod \"e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr\" (UID: \"4659fd62-58a7-4792-ad28-93f3c52c7d3a\") " pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr" Nov 01 20:17:29.972560 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:29.972543 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pv55\" (UniqueName: \"kubernetes.io/projected/4659fd62-58a7-4792-ad28-93f3c52c7d3a-kube-api-access-4pv55\") pod \"e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr\" (UID: \"4659fd62-58a7-4792-ad28-93f3c52c7d3a\") " pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr" Nov 01 20:17:30.090785 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:30.090679 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr" Nov 01 20:17:30.271027 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:30.270978 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bb8ccaa1-ea3f-4f91-ad5d-173a28d77459-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-849d6msts\" (UID: \"bb8ccaa1-ea3f-4f91-ad5d-173a28d77459\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-849d6msts" Nov 01 20:17:30.271198 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:17:30.271172 5935 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-aaea41/agc-cred-b2b285556e64: secret "agc-cred-b2b285556e64" not found Nov 01 20:17:30.271279 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:17:30.271209 5935 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-849d6msts: secret "agc-cred-b2b285556e64" not found Nov 01 20:17:30.271389 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:17:30.271322 5935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bb8ccaa1-ea3f-4f91-ad5d-173a28d77459-provider-credentials podName:bb8ccaa1-ea3f-4f91-ad5d-173a28d77459 nodeName:}" failed. No retries permitted until 2026-11-01 20:17:32.271268728 +0000 UTC m=+1098.044429231 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/bb8ccaa1-ea3f-4f91-ad5d-173a28d77459-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-aaea41-849d6msts" (UID: "bb8ccaa1-ea3f-4f91-ad5d-173a28d77459") : secret "agc-cred-b2b285556e64" not found Nov 01 20:17:30.384074 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:30.384048 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr"] Nov 01 20:17:30.384684 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:17:30.384659 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4659fd62_58a7_4792_ad28_93f3c52c7d3a.slice/crio-4b92b864859bce30daf499d2bf456248ad5a67801c762de07127833dfac5cdea WatchSource:0}: Error finding container 4b92b864859bce30daf499d2bf456248ad5a67801c762de07127833dfac5cdea: Status 404 returned error can't find the container with id 4b92b864859bce30daf499d2bf456248ad5a67801c762de07127833dfac5cdea Nov 01 20:17:30.387299 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:30.387249 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:17:30.387362 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:30.387348 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:17:30.387407 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:30.387395 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:17:30.787436 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:30.786256 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr" event={"ID":"4659fd62-58a7-4792-ad28-93f3c52c7d3a","Type":"ContainerStarted","Data":"1a27da7c75ee3916ada810397f40cd896f05191e29efe85b369242c70522e24c"} Nov 01 20:17:30.787436 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:30.786360 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr" event={"ID":"4659fd62-58a7-4792-ad28-93f3c52c7d3a","Type":"ContainerStarted","Data":"4b92b864859bce30daf499d2bf456248ad5a67801c762de07127833dfac5cdea"} Nov 01 20:17:31.090948 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:31.090862 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr" Nov 01 20:17:31.092396 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:31.092368 5935 patch_prober.go:29] interesting pod/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 01 20:17:31.092514 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:31.092431 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr" podUID="4659fd62-58a7-4792-ad28-93f3c52c7d3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 01 20:17:32.091944 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:32.091908 5935 patch_prober.go:29] interesting pod/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 01 20:17:32.092382 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:32.091975 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr" podUID="4659fd62-58a7-4792-ad28-93f3c52c7d3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 01 20:17:32.301985 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:32.301951 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bb8ccaa1-ea3f-4f91-ad5d-173a28d77459-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-849d6msts\" (UID: \"bb8ccaa1-ea3f-4f91-ad5d-173a28d77459\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-849d6msts" Nov 01 20:17:32.302174 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:17:32.302109 5935 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-aaea41/agc-cred-b2b285556e64: secret "agc-cred-b2b285556e64" not found Nov 01 20:17:32.302174 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:17:32.302131 5935 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-849d6msts: secret "agc-cred-b2b285556e64" not found Nov 01 20:17:32.302346 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:17:32.302188 5935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bb8ccaa1-ea3f-4f91-ad5d-173a28d77459-provider-credentials podName:bb8ccaa1-ea3f-4f91-ad5d-173a28d77459 nodeName:}" failed. No retries permitted until 2026-11-01 20:17:36.30217042 +0000 UTC m=+1102.075330923 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/bb8ccaa1-ea3f-4f91-ad5d-173a28d77459-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-aaea41-849d6msts" (UID: "bb8ccaa1-ea3f-4f91-ad5d-173a28d77459") : secret "agc-cred-b2b285556e64" not found Nov 01 20:17:33.091375 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:33.091339 5935 patch_prober.go:29] interesting pod/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 01 20:17:33.091587 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:33.091398 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr" podUID="4659fd62-58a7-4792-ad28-93f3c52c7d3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 01 20:17:34.091961 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:34.091922 5935 patch_prober.go:29] interesting pod/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 01 20:17:34.092415 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:34.091983 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr" podUID="4659fd62-58a7-4792-ad28-93f3c52c7d3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 01 20:17:35.091203 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:35.091169 5935 patch_prober.go:29] interesting pod/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 01 20:17:35.091420 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:35.091228 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr" podUID="4659fd62-58a7-4792-ad28-93f3c52c7d3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 01 20:17:35.881747 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:17:35.881709 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:17:36.091837 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:36.091757 5935 patch_prober.go:29] interesting pod/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Nov 01 20:17:36.091837 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:36.091813 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr" podUID="4659fd62-58a7-4792-ad28-93f3c52c7d3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Nov 01 20:17:36.307813 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:36.307784 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bb8ccaa1-ea3f-4f91-ad5d-173a28d77459-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-849d6msts\" (UID: \"bb8ccaa1-ea3f-4f91-ad5d-173a28d77459\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-849d6msts" Nov 01 20:17:36.307975 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:17:36.307943 5935 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-aaea41/agc-cred-b2b285556e64: secret "agc-cred-b2b285556e64" not found Nov 01 20:17:36.307975 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:17:36.307968 5935 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-849d6msts: secret "agc-cred-b2b285556e64" not found Nov 01 20:17:36.308049 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:17:36.308030 5935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bb8ccaa1-ea3f-4f91-ad5d-173a28d77459-provider-credentials podName:bb8ccaa1-ea3f-4f91-ad5d-173a28d77459 nodeName:}" failed. No retries permitted until 2026-11-01 20:17:44.308016054 +0000 UTC m=+1110.081176560 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/bb8ccaa1-ea3f-4f91-ad5d-173a28d77459-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-aaea41-849d6msts" (UID: "bb8ccaa1-ea3f-4f91-ad5d-173a28d77459") : secret "agc-cred-b2b285556e64" not found Nov 01 20:17:37.169881 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:37.169850 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr" Nov 01 20:17:37.170970 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:37.170950 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr" Nov 01 20:17:37.171853 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:37.171835 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr" Nov 01 20:17:37.191744 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:37.191699 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr" podStartSLOduration=8.191687955999999 podStartE2EDuration="8.191687956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:17:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:17:30.810407654 +0000 UTC m=+1096.583568166" watchObservedRunningTime="2026-11-01 20:17:37.191687956 +0000 UTC m=+1102.964848468" Nov 01 20:17:41.167073 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:41.167038 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-849d6msts"] Nov 01 20:17:41.167688 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:17:41.167352 5935 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-849d6msts" podUID="bb8ccaa1-ea3f-4f91-ad5d-173a28d77459" Nov 01 20:17:41.943860 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:41.943830 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-849d6msts" Nov 01 20:17:41.956711 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:41.956687 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-849d6msts" Nov 01 20:17:41.980354 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:41.980323 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb8ccaa1-ea3f-4f91-ad5d-173a28d77459-kube-api-access-kpkch\" (UniqueName: \"kubernetes.io/projected/bb8ccaa1-ea3f-4f91-ad5d-173a28d77459-kube-api-access-kpkch\") pod \"bb8ccaa1-ea3f-4f91-ad5d-173a28d77459\" (UID: \"bb8ccaa1-ea3f-4f91-ad5d-173a28d77459\") " Nov 01 20:17:41.980551 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:41.980400 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb8ccaa1-ea3f-4f91-ad5d-173a28d77459-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb8ccaa1-ea3f-4f91-ad5d-173a28d77459-plugins-config-volume\") pod \"bb8ccaa1-ea3f-4f91-ad5d-173a28d77459\" (UID: \"bb8ccaa1-ea3f-4f91-ad5d-173a28d77459\") " Nov 01 20:17:41.980551 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:41.980476 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb8ccaa1-ea3f-4f91-ad5d-173a28d77459-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bb8ccaa1-ea3f-4f91-ad5d-173a28d77459-routing-overlay-volume\") pod \"bb8ccaa1-ea3f-4f91-ad5d-173a28d77459\" (UID: \"bb8ccaa1-ea3f-4f91-ad5d-173a28d77459\") " Nov 01 20:17:41.980795 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:41.980771 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb8ccaa1-ea3f-4f91-ad5d-173a28d77459-routing-overlay-volume" pod "bb8ccaa1-ea3f-4f91-ad5d-173a28d77459" (UID: "bb8ccaa1-ea3f-4f91-ad5d-173a28d77459"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:17:41.980939 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:41.980914 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb8ccaa1-ea3f-4f91-ad5d-173a28d77459-plugins-config-volume" pod "bb8ccaa1-ea3f-4f91-ad5d-173a28d77459" (UID: "bb8ccaa1-ea3f-4f91-ad5d-173a28d77459"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:17:41.982985 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:41.982960 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb8ccaa1-ea3f-4f91-ad5d-173a28d77459-kube-api-access-kpkch" pod "bb8ccaa1-ea3f-4f91-ad5d-173a28d77459" (UID: "bb8ccaa1-ea3f-4f91-ad5d-173a28d77459"). InnerVolumeSpecName "kube-api-access-kpkch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:17:42.082201 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:42.082168 5935 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bb8ccaa1-ea3f-4f91-ad5d-173a28d77459-routing-overlay-volume\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:17:42.082201 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:42.082194 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpkch\" (UniqueName: \"kubernetes.io/projected/bb8ccaa1-ea3f-4f91-ad5d-173a28d77459-kube-api-access-kpkch\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:17:42.082201 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:42.082204 5935 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb8ccaa1-ea3f-4f91-ad5d-173a28d77459-plugins-config-volume\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:17:42.966078 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:42.961307 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-849d6msts" Nov 01 20:17:43.028391 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:43.028356 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-849d6msts"] Nov 01 20:17:43.033881 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:43.033854 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-849d6msts"] Nov 01 20:17:43.096179 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:43.096149 5935 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bb8ccaa1-ea3f-4f91-ad5d-173a28d77459-provider-credentials\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:17:44.354526 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:44.354490 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb8ccaa1-ea3f-4f91-ad5d-173a28d77459" path="/var/lib/kubelet/pods/bb8ccaa1-ea3f-4f91-ad5d-173a28d77459/volumes" Nov 01 20:17:49.179207 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:49.179176 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6cfdjr9s4"] Nov 01 20:17:49.184489 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:49.184469 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6cfdjr9s4" Nov 01 20:17:49.188946 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:49.188924 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-bccce1\"/\"payload-processing-external-model-plugins-e2e-extmod-b-bccce1\"" Nov 01 20:17:49.194546 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:49.190021 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-bccce1\"/\"routing-overlay\"" Nov 01 20:17:49.194546 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:49.190060 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-bccce1\"/\"openshift-service-ca.crt\"" Nov 01 20:17:49.194546 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:49.190092 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-bccce1\"/\"agc-cred-fec91d8db4b3\"" Nov 01 20:17:49.194546 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:49.190138 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-bccce1\"/\"kube-root-ca.crt\"" Nov 01 20:17:49.194546 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:49.190143 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-bccce1\"/\"agc-cred-85391858a611\"" Nov 01 20:17:49.194546 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:49.190143 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-bccce1\"/\"agc-cred-646b8d4cafa4\"" Nov 01 20:17:49.194546 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:49.190454 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-bccce1\"/\"payload-processing-external-model-e2e--900891f9-dockercfg-ns4ld\"" Nov 01 20:17:49.194546 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:49.194500 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6cfdjr9s4"] Nov 01 20:17:49.278487 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:49.278452 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jklsp\" (UniqueName: \"kubernetes.io/projected/0e43111a-8bdd-4d15-8ec5-be5ee2b8d194-kube-api-access-jklsp\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-6cfdjr9s4\" (UID: \"0e43111a-8bdd-4d15-8ec5-be5ee2b8d194\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6cfdjr9s4" Nov 01 20:17:49.278996 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:49.278955 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e43111a-8bdd-4d15-8ec5-be5ee2b8d194-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-6cfdjr9s4\" (UID: \"0e43111a-8bdd-4d15-8ec5-be5ee2b8d194\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6cfdjr9s4" Nov 01 20:17:49.279142 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:49.279113 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0e43111a-8bdd-4d15-8ec5-be5ee2b8d194-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-6cfdjr9s4\" (UID: \"0e43111a-8bdd-4d15-8ec5-be5ee2b8d194\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6cfdjr9s4" Nov 01 20:17:49.279413 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:49.279397 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0e43111a-8bdd-4d15-8ec5-be5ee2b8d194-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-6cfdjr9s4\" (UID: \"0e43111a-8bdd-4d15-8ec5-be5ee2b8d194\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6cfdjr9s4" Nov 01 20:17:49.381421 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:49.381383 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e43111a-8bdd-4d15-8ec5-be5ee2b8d194-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-6cfdjr9s4\" (UID: \"0e43111a-8bdd-4d15-8ec5-be5ee2b8d194\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6cfdjr9s4" Nov 01 20:17:49.381421 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:49.381423 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0e43111a-8bdd-4d15-8ec5-be5ee2b8d194-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-6cfdjr9s4\" (UID: \"0e43111a-8bdd-4d15-8ec5-be5ee2b8d194\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6cfdjr9s4" Nov 01 20:17:49.381725 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:49.381563 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0e43111a-8bdd-4d15-8ec5-be5ee2b8d194-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-6cfdjr9s4\" (UID: \"0e43111a-8bdd-4d15-8ec5-be5ee2b8d194\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6cfdjr9s4" Nov 01 20:17:49.381725 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:49.381646 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jklsp\" (UniqueName: \"kubernetes.io/projected/0e43111a-8bdd-4d15-8ec5-be5ee2b8d194-kube-api-access-jklsp\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-6cfdjr9s4\" (UID: \"0e43111a-8bdd-4d15-8ec5-be5ee2b8d194\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6cfdjr9s4" Nov 01 20:17:49.382509 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:49.382481 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0e43111a-8bdd-4d15-8ec5-be5ee2b8d194-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-6cfdjr9s4\" (UID: \"0e43111a-8bdd-4d15-8ec5-be5ee2b8d194\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6cfdjr9s4" Nov 01 20:17:49.382863 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:49.382844 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e43111a-8bdd-4d15-8ec5-be5ee2b8d194-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-6cfdjr9s4\" (UID: \"0e43111a-8bdd-4d15-8ec5-be5ee2b8d194\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6cfdjr9s4" Nov 01 20:17:49.384565 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:49.384542 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0e43111a-8bdd-4d15-8ec5-be5ee2b8d194-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-6cfdjr9s4\" (UID: \"0e43111a-8bdd-4d15-8ec5-be5ee2b8d194\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6cfdjr9s4" Nov 01 20:17:49.389934 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:49.389905 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jklsp\" (UniqueName: \"kubernetes.io/projected/0e43111a-8bdd-4d15-8ec5-be5ee2b8d194-kube-api-access-jklsp\") pod \"payload-processing-external-model-e2e-extmod-b-bccce1-6cfdjr9s4\" (UID: \"0e43111a-8bdd-4d15-8ec5-be5ee2b8d194\") " pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6cfdjr9s4" Nov 01 20:17:49.500263 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:49.500235 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6cfdjr9s4" Nov 01 20:17:49.801498 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:49.801468 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6cfdjr9s4"] Nov 01 20:17:49.802141 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:17:49.802113 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e43111a_8bdd_4d15_8ec5_be5ee2b8d194.slice/crio-ca41443a0f8f0fa84caae8ab74cdb51bf88edaf4dab995947ec457f93230e765 WatchSource:0}: Error finding container ca41443a0f8f0fa84caae8ab74cdb51bf88edaf4dab995947ec457f93230e765: Status 404 returned error can't find the container with id ca41443a0f8f0fa84caae8ab74cdb51bf88edaf4dab995947ec457f93230e765 Nov 01 20:17:50.017688 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:50.017652 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6cfdjr9s4" event={"ID":"0e43111a-8bdd-4d15-8ec5-be5ee2b8d194","Type":"ContainerStarted","Data":"0f9dcbe4607701d972758c16472e95d13a10129c1f1a5703f76172689bf01ee4"} Nov 01 20:17:50.017688 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:50.017708 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6cfdjr9s4" event={"ID":"0e43111a-8bdd-4d15-8ec5-be5ee2b8d194","Type":"ContainerStarted","Data":"ca41443a0f8f0fa84caae8ab74cdb51bf88edaf4dab995947ec457f93230e765"} Nov 01 20:17:50.017949 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:50.017731 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6cfdjr9s4" Nov 01 20:17:50.033446 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:50.033396 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6cfdjr9s4" podStartSLOduration=1.033384173 podStartE2EDuration="1.033384173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:17:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:17:50.032055183 +0000 UTC m=+1115.805215707" watchObservedRunningTime="2026-11-01 20:17:50.033384173 +0000 UTC m=+1115.806544684" Nov 01 20:17:51.104699 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:17:51.104663 5935 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-bccce1/agc-cred-fec91d8db4b3: secret "agc-cred-fec91d8db4b3" not found Nov 01 20:17:51.105235 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:17:51.105217 5935 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6cfdjr9s4: secret "agc-cred-fec91d8db4b3" not found Nov 01 20:17:51.105430 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:17:51.105415 5935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0e43111a-8bdd-4d15-8ec5-be5ee2b8d194-provider-credentials podName:0e43111a-8bdd-4d15-8ec5-be5ee2b8d194 nodeName:}" failed. No retries permitted until 2026-11-01 20:17:51.605391139 +0000 UTC m=+1117.378551632 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0e43111a-8bdd-4d15-8ec5-be5ee2b8d194-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-bccce1-6cfdjr9s4" (UID: "0e43111a-8bdd-4d15-8ec5-be5ee2b8d194") : secret "agc-cred-fec91d8db4b3" not found Nov 01 20:17:51.471894 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:51.471856 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6cfdjr9s4"] Nov 01 20:17:51.658443 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:17:51.658405 5935 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-bccce1/agc-cred-fec91d8db4b3: secret "agc-cred-fec91d8db4b3" not found Nov 01 20:17:51.658677 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:17:51.658661 5935 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6cfdjr9s4: secret "agc-cred-fec91d8db4b3" not found Nov 01 20:17:51.658850 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:17:51.658837 5935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0e43111a-8bdd-4d15-8ec5-be5ee2b8d194-provider-credentials podName:0e43111a-8bdd-4d15-8ec5-be5ee2b8d194 nodeName:}" failed. No retries permitted until 2026-11-01 20:17:52.658813663 +0000 UTC m=+1118.431974153 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0e43111a-8bdd-4d15-8ec5-be5ee2b8d194-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-bccce1-6cfdjr9s4" (UID: "0e43111a-8bdd-4d15-8ec5-be5ee2b8d194") : secret "agc-cred-fec91d8db4b3" not found Nov 01 20:17:52.670384 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:17:52.670355 5935 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-bccce1/agc-cred-fec91d8db4b3: secret "agc-cred-fec91d8db4b3" not found Nov 01 20:17:52.670384 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:17:52.670380 5935 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6cfdjr9s4: secret "agc-cred-fec91d8db4b3" not found Nov 01 20:17:52.670894 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:17:52.670456 5935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0e43111a-8bdd-4d15-8ec5-be5ee2b8d194-provider-credentials podName:0e43111a-8bdd-4d15-8ec5-be5ee2b8d194 nodeName:}" failed. No retries permitted until 2026-11-01 20:17:54.670436726 +0000 UTC m=+1120.443597229 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0e43111a-8bdd-4d15-8ec5-be5ee2b8d194-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-bccce1-6cfdjr9s4" (UID: "0e43111a-8bdd-4d15-8ec5-be5ee2b8d194") : secret "agc-cred-fec91d8db4b3" not found Nov 01 20:17:54.697886 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:17:54.697851 5935 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-bccce1/agc-cred-fec91d8db4b3: secret "agc-cred-fec91d8db4b3" not found Nov 01 20:17:54.697886 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:17:54.697877 5935 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6cfdjr9s4: secret "agc-cred-fec91d8db4b3" not found Nov 01 20:17:54.698360 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:17:54.697929 5935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0e43111a-8bdd-4d15-8ec5-be5ee2b8d194-provider-credentials podName:0e43111a-8bdd-4d15-8ec5-be5ee2b8d194 nodeName:}" failed. No retries permitted until 2026-11-01 20:17:58.697914995 +0000 UTC m=+1124.471075490 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/0e43111a-8bdd-4d15-8ec5-be5ee2b8d194-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-bccce1-6cfdjr9s4" (UID: "0e43111a-8bdd-4d15-8ec5-be5ee2b8d194") : secret "agc-cred-fec91d8db4b3" not found Nov 01 20:17:57.032867 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:57.032810 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6cfdjr9s4" podUID="0e43111a-8bdd-4d15-8ec5-be5ee2b8d194" containerName="payload-processing" containerID="cri-o://0f9dcbe4607701d972758c16472e95d13a10129c1f1a5703f76172689bf01ee4" gracePeriod=25 Nov 01 20:17:57.313048 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:57.313025 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6cfdjr9s4" Nov 01 20:17:57.431848 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:57.431816 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e43111a-8bdd-4d15-8ec5-be5ee2b8d194-kube-api-access-jklsp\" (UniqueName: \"kubernetes.io/projected/0e43111a-8bdd-4d15-8ec5-be5ee2b8d194-kube-api-access-jklsp\") pod \"0e43111a-8bdd-4d15-8ec5-be5ee2b8d194\" (UID: \"0e43111a-8bdd-4d15-8ec5-be5ee2b8d194\") " Nov 01 20:17:57.432023 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:57.431915 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e43111a-8bdd-4d15-8ec5-be5ee2b8d194-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0e43111a-8bdd-4d15-8ec5-be5ee2b8d194-routing-overlay-volume\") pod \"0e43111a-8bdd-4d15-8ec5-be5ee2b8d194\" (UID: \"0e43111a-8bdd-4d15-8ec5-be5ee2b8d194\") " Nov 01 20:17:57.432023 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:57.431988 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e43111a-8bdd-4d15-8ec5-be5ee2b8d194-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e43111a-8bdd-4d15-8ec5-be5ee2b8d194-plugins-config-volume\") pod \"0e43111a-8bdd-4d15-8ec5-be5ee2b8d194\" (UID: \"0e43111a-8bdd-4d15-8ec5-be5ee2b8d194\") " Nov 01 20:17:57.432204 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:57.432032 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e43111a-8bdd-4d15-8ec5-be5ee2b8d194-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0e43111a-8bdd-4d15-8ec5-be5ee2b8d194-provider-credentials\") pod \"0e43111a-8bdd-4d15-8ec5-be5ee2b8d194\" (UID: \"0e43111a-8bdd-4d15-8ec5-be5ee2b8d194\") " Nov 01 20:17:57.432353 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:57.432323 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e43111a-8bdd-4d15-8ec5-be5ee2b8d194-routing-overlay-volume" pod "0e43111a-8bdd-4d15-8ec5-be5ee2b8d194" (UID: "0e43111a-8bdd-4d15-8ec5-be5ee2b8d194"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:17:57.432439 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:57.432413 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e43111a-8bdd-4d15-8ec5-be5ee2b8d194-plugins-config-volume" pod "0e43111a-8bdd-4d15-8ec5-be5ee2b8d194" (UID: "0e43111a-8bdd-4d15-8ec5-be5ee2b8d194"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:17:57.432668 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:57.432638 5935 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0e43111a-8bdd-4d15-8ec5-be5ee2b8d194-routing-overlay-volume\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:17:57.432668 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:57.432670 5935 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0e43111a-8bdd-4d15-8ec5-be5ee2b8d194-plugins-config-volume\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:17:57.434373 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:57.434345 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e43111a-8bdd-4d15-8ec5-be5ee2b8d194-kube-api-access-jklsp" pod "0e43111a-8bdd-4d15-8ec5-be5ee2b8d194" (UID: "0e43111a-8bdd-4d15-8ec5-be5ee2b8d194"). InnerVolumeSpecName "kube-api-access-jklsp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:17:57.434461 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:57.434401 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e43111a-8bdd-4d15-8ec5-be5ee2b8d194-provider-credentials" pod "0e43111a-8bdd-4d15-8ec5-be5ee2b8d194" (UID: "0e43111a-8bdd-4d15-8ec5-be5ee2b8d194"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:17:57.533472 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:57.533442 5935 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0e43111a-8bdd-4d15-8ec5-be5ee2b8d194-provider-credentials\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:17:57.533472 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:57.533468 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jklsp\" (UniqueName: \"kubernetes.io/projected/0e43111a-8bdd-4d15-8ec5-be5ee2b8d194-kube-api-access-jklsp\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:17:58.073904 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:58.073866 5935 generic.go:356] "Generic (PLEG): container finished" podID="0e43111a-8bdd-4d15-8ec5-be5ee2b8d194" containerID="0f9dcbe4607701d972758c16472e95d13a10129c1f1a5703f76172689bf01ee4" exitCode=0 Nov 01 20:17:58.074427 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:58.073915 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6cfdjr9s4" Nov 01 20:17:58.074427 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:58.073956 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6cfdjr9s4" event={"ID":"0e43111a-8bdd-4d15-8ec5-be5ee2b8d194","Type":"ContainerDied","Data":"0f9dcbe4607701d972758c16472e95d13a10129c1f1a5703f76172689bf01ee4"} Nov 01 20:17:58.074427 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:58.074014 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6cfdjr9s4" event={"ID":"0e43111a-8bdd-4d15-8ec5-be5ee2b8d194","Type":"ContainerDied","Data":"ca41443a0f8f0fa84caae8ab74cdb51bf88edaf4dab995947ec457f93230e765"} Nov 01 20:17:58.074427 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:58.074051 5935 scope.go:122] "RemoveContainer" containerID="0f9dcbe4607701d972758c16472e95d13a10129c1f1a5703f76172689bf01ee4" Nov 01 20:17:58.084534 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:58.084519 5935 scope.go:122] "RemoveContainer" containerID="0f9dcbe4607701d972758c16472e95d13a10129c1f1a5703f76172689bf01ee4" Nov 01 20:17:58.084805 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:17:58.084786 5935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f9dcbe4607701d972758c16472e95d13a10129c1f1a5703f76172689bf01ee4\": container with ID starting with 0f9dcbe4607701d972758c16472e95d13a10129c1f1a5703f76172689bf01ee4 not found: ID does not exist" containerID="0f9dcbe4607701d972758c16472e95d13a10129c1f1a5703f76172689bf01ee4" Nov 01 20:17:58.084870 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:58.084817 5935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f9dcbe4607701d972758c16472e95d13a10129c1f1a5703f76172689bf01ee4"} err="failed to get container status \"0f9dcbe4607701d972758c16472e95d13a10129c1f1a5703f76172689bf01ee4\": rpc error: code = NotFound desc = could not find container \"0f9dcbe4607701d972758c16472e95d13a10129c1f1a5703f76172689bf01ee4\": container with ID starting with 0f9dcbe4607701d972758c16472e95d13a10129c1f1a5703f76172689bf01ee4 not found: ID does not exist" Nov 01 20:17:58.096246 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:58.096224 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6cfdjr9s4"] Nov 01 20:17:58.099434 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:58.099414 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-bccce1/payload-processing-external-model-e2e-extmod-b-bccce1-6cfdjr9s4"] Nov 01 20:17:58.357061 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:17:58.356976 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e43111a-8bdd-4d15-8ec5-be5ee2b8d194" path="/var/lib/kubelet/pods/0e43111a-8bdd-4d15-8ec5-be5ee2b8d194/volumes" Nov 01 20:18:05.844222 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:18:05.844188 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:18:09.558632 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:09.558594 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-68f76b7b94-ghwhj"] Nov 01 20:18:09.559351 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:09.559335 5935 cpu_manager.go:414] "containerMap: removing container" podUID="0e43111a-8bdd-4d15-8ec5-be5ee2b8d194" containerName="payload-processing" Nov 01 20:18:09.559417 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:09.559353 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e43111a-8bdd-4d15-8ec5-be5ee2b8d194" containerName="payload-processing" Nov 01 20:18:09.559458 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:09.559442 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e43111a-8bdd-4d15-8ec5-be5ee2b8d194" containerName="payload-processing" Nov 01 20:18:09.563018 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:09.563003 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-68f76b7b94-ghwhj" Nov 01 20:18:09.570914 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:09.570738 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-1423bbb43273\"" Nov 01 20:18:09.570914 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:09.570738 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 01 20:18:09.572171 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:09.572064 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 01 20:18:09.572171 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:09.572117 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-fcf37fb5bad9\"" Nov 01 20:18:09.572171 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:09.572140 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 01 20:18:09.572426 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:09.572374 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-g8lzb\"" Nov 01 20:18:09.574373 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:09.574343 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 01 20:18:09.579239 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:09.576541 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-68f76b7b94-ghwhj"] Nov 01 20:18:09.693833 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:09.693792 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c293efa4-25da-468d-8f4e-12ebd296a217-provider-credentials\") pod \"payload-processing-external-model-68f76b7b94-ghwhj\" (UID: \"c293efa4-25da-468d-8f4e-12ebd296a217\") " pod="models-as-a-service/payload-processing-external-model-68f76b7b94-ghwhj" Nov 01 20:18:09.694052 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:09.693956 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmjhg\" (UniqueName: \"kubernetes.io/projected/c293efa4-25da-468d-8f4e-12ebd296a217-kube-api-access-rmjhg\") pod \"payload-processing-external-model-68f76b7b94-ghwhj\" (UID: \"c293efa4-25da-468d-8f4e-12ebd296a217\") " pod="models-as-a-service/payload-processing-external-model-68f76b7b94-ghwhj" Nov 01 20:18:09.694052 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:09.694008 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c293efa4-25da-468d-8f4e-12ebd296a217-plugins-config-volume\") pod \"payload-processing-external-model-68f76b7b94-ghwhj\" (UID: \"c293efa4-25da-468d-8f4e-12ebd296a217\") " pod="models-as-a-service/payload-processing-external-model-68f76b7b94-ghwhj" Nov 01 20:18:09.694174 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:09.694076 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c293efa4-25da-468d-8f4e-12ebd296a217-routing-overlay-volume\") pod \"payload-processing-external-model-68f76b7b94-ghwhj\" (UID: \"c293efa4-25da-468d-8f4e-12ebd296a217\") " pod="models-as-a-service/payload-processing-external-model-68f76b7b94-ghwhj" Nov 01 20:18:09.797255 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:09.797211 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c293efa4-25da-468d-8f4e-12ebd296a217-provider-credentials\") pod \"payload-processing-external-model-68f76b7b94-ghwhj\" (UID: \"c293efa4-25da-468d-8f4e-12ebd296a217\") " pod="models-as-a-service/payload-processing-external-model-68f76b7b94-ghwhj" Nov 01 20:18:09.797697 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:09.797670 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmjhg\" (UniqueName: \"kubernetes.io/projected/c293efa4-25da-468d-8f4e-12ebd296a217-kube-api-access-rmjhg\") pod \"payload-processing-external-model-68f76b7b94-ghwhj\" (UID: \"c293efa4-25da-468d-8f4e-12ebd296a217\") " pod="models-as-a-service/payload-processing-external-model-68f76b7b94-ghwhj" Nov 01 20:18:09.797895 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:09.797877 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c293efa4-25da-468d-8f4e-12ebd296a217-plugins-config-volume\") pod \"payload-processing-external-model-68f76b7b94-ghwhj\" (UID: \"c293efa4-25da-468d-8f4e-12ebd296a217\") " pod="models-as-a-service/payload-processing-external-model-68f76b7b94-ghwhj" Nov 01 20:18:09.798070 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:09.798053 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c293efa4-25da-468d-8f4e-12ebd296a217-routing-overlay-volume\") pod \"payload-processing-external-model-68f76b7b94-ghwhj\" (UID: \"c293efa4-25da-468d-8f4e-12ebd296a217\") " pod="models-as-a-service/payload-processing-external-model-68f76b7b94-ghwhj" Nov 01 20:18:09.809353 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:09.799074 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c293efa4-25da-468d-8f4e-12ebd296a217-routing-overlay-volume\") pod \"payload-processing-external-model-68f76b7b94-ghwhj\" (UID: \"c293efa4-25da-468d-8f4e-12ebd296a217\") " pod="models-as-a-service/payload-processing-external-model-68f76b7b94-ghwhj" Nov 01 20:18:09.823249 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:09.823215 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmjhg\" (UniqueName: \"kubernetes.io/projected/c293efa4-25da-468d-8f4e-12ebd296a217-kube-api-access-rmjhg\") pod \"payload-processing-external-model-68f76b7b94-ghwhj\" (UID: \"c293efa4-25da-468d-8f4e-12ebd296a217\") " pod="models-as-a-service/payload-processing-external-model-68f76b7b94-ghwhj" Nov 01 20:18:09.825499 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:09.825468 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c293efa4-25da-468d-8f4e-12ebd296a217-plugins-config-volume\") pod \"payload-processing-external-model-68f76b7b94-ghwhj\" (UID: \"c293efa4-25da-468d-8f4e-12ebd296a217\") " pod="models-as-a-service/payload-processing-external-model-68f76b7b94-ghwhj" Nov 01 20:18:09.837328 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:09.837281 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c293efa4-25da-468d-8f4e-12ebd296a217-provider-credentials\") pod \"payload-processing-external-model-68f76b7b94-ghwhj\" (UID: \"c293efa4-25da-468d-8f4e-12ebd296a217\") " pod="models-as-a-service/payload-processing-external-model-68f76b7b94-ghwhj" Nov 01 20:18:09.882121 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:09.882086 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-68f76b7b94-ghwhj" Nov 01 20:18:10.176048 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:10.176021 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-68f76b7b94-ghwhj"] Nov 01 20:18:10.176642 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:18:10.176620 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc293efa4_25da_468d_8f4e_12ebd296a217.slice/crio-2152093139c88e75cd5458bbac3c5e291653caa06028901e68c3b1f695ceba1e WatchSource:0}: Error finding container 2152093139c88e75cd5458bbac3c5e291653caa06028901e68c3b1f695ceba1e: Status 404 returned error can't find the container with id 2152093139c88e75cd5458bbac3c5e291653caa06028901e68c3b1f695ceba1e Nov 01 20:18:11.172090 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:11.172056 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-68f76b7b94-ghwhj" event={"ID":"c293efa4-25da-468d-8f4e-12ebd296a217","Type":"ContainerStarted","Data":"9546eee773590057ead2ed95e2bc095de5767800e713ca0477366e84566432cd"} Nov 01 20:18:11.172530 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:11.172113 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-68f76b7b94-ghwhj" Nov 01 20:18:11.172530 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:11.172129 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-68f76b7b94-ghwhj" event={"ID":"c293efa4-25da-468d-8f4e-12ebd296a217","Type":"ContainerStarted","Data":"2152093139c88e75cd5458bbac3c5e291653caa06028901e68c3b1f695ceba1e"} Nov 01 20:18:11.191144 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:11.191096 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-68f76b7b94-ghwhj" podStartSLOduration=2.191083712 podStartE2EDuration="2.191083712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:18:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:18:11.189053197 +0000 UTC m=+1136.962213711" watchObservedRunningTime="2026-11-01 20:18:11.191083712 +0000 UTC m=+1136.964244242" Nov 01 20:18:13.967617 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:13.967586 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-799fnb2vq"] Nov 01 20:18:13.971735 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:13.971718 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-799fnb2vq" Nov 01 20:18:13.980432 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:13.980390 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-29b57c\"/\"kube-root-ca.crt\"" Nov 01 20:18:13.980432 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:13.980423 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-29b57c\"/\"payload-processing-external-model-plugins-e2e-extmod-a-29b57c\"" Nov 01 20:18:13.980618 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:13.980432 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-29b57c\"/\"payload-processing-external-model-e2e--80e6ed0f-dockercfg-9xw72\"" Nov 01 20:18:13.980618 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:13.980443 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-29b57c\"/\"agc-cred-6da735d022c1\"" Nov 01 20:18:13.980618 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:13.980450 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-29b57c\"/\"routing-overlay\"" Nov 01 20:18:13.980618 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:13.980466 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-29b57c\"/\"openshift-service-ca.crt\"" Nov 01 20:18:13.980759 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:13.980651 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-799fnb2vq"] Nov 01 20:18:14.107807 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:14.107768 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8540f99e-96ca-45f5-922a-2a640d2c9d72-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-799fnb2vq\" (UID: \"8540f99e-96ca-45f5-922a-2a640d2c9d72\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-799fnb2vq" Nov 01 20:18:14.108014 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:14.107841 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8540f99e-96ca-45f5-922a-2a640d2c9d72-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-799fnb2vq\" (UID: \"8540f99e-96ca-45f5-922a-2a640d2c9d72\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-799fnb2vq" Nov 01 20:18:14.108117 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:14.108101 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-788ph\" (UniqueName: \"kubernetes.io/projected/8540f99e-96ca-45f5-922a-2a640d2c9d72-kube-api-access-788ph\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-799fnb2vq\" (UID: \"8540f99e-96ca-45f5-922a-2a640d2c9d72\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-799fnb2vq" Nov 01 20:18:14.108184 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:14.108165 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8540f99e-96ca-45f5-922a-2a640d2c9d72-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-799fnb2vq\" (UID: \"8540f99e-96ca-45f5-922a-2a640d2c9d72\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-799fnb2vq" Nov 01 20:18:14.209431 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:14.209401 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-788ph\" (UniqueName: \"kubernetes.io/projected/8540f99e-96ca-45f5-922a-2a640d2c9d72-kube-api-access-788ph\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-799fnb2vq\" (UID: \"8540f99e-96ca-45f5-922a-2a640d2c9d72\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-799fnb2vq" Nov 01 20:18:14.209607 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:14.209463 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8540f99e-96ca-45f5-922a-2a640d2c9d72-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-799fnb2vq\" (UID: \"8540f99e-96ca-45f5-922a-2a640d2c9d72\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-799fnb2vq" Nov 01 20:18:14.209607 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:14.209533 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8540f99e-96ca-45f5-922a-2a640d2c9d72-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-799fnb2vq\" (UID: \"8540f99e-96ca-45f5-922a-2a640d2c9d72\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-799fnb2vq" Nov 01 20:18:14.209607 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:14.209581 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8540f99e-96ca-45f5-922a-2a640d2c9d72-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-799fnb2vq\" (UID: \"8540f99e-96ca-45f5-922a-2a640d2c9d72\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-799fnb2vq" Nov 01 20:18:14.210390 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:14.210365 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8540f99e-96ca-45f5-922a-2a640d2c9d72-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-799fnb2vq\" (UID: \"8540f99e-96ca-45f5-922a-2a640d2c9d72\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-799fnb2vq" Nov 01 20:18:14.210522 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:14.210433 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8540f99e-96ca-45f5-922a-2a640d2c9d72-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-799fnb2vq\" (UID: \"8540f99e-96ca-45f5-922a-2a640d2c9d72\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-799fnb2vq" Nov 01 20:18:14.212095 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:14.212074 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8540f99e-96ca-45f5-922a-2a640d2c9d72-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-799fnb2vq\" (UID: \"8540f99e-96ca-45f5-922a-2a640d2c9d72\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-799fnb2vq" Nov 01 20:18:14.216851 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:14.216829 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-788ph\" (UniqueName: \"kubernetes.io/projected/8540f99e-96ca-45f5-922a-2a640d2c9d72-kube-api-access-788ph\") pod \"payload-processing-external-model-e2e-extmod-a-29b57c-799fnb2vq\" (UID: \"8540f99e-96ca-45f5-922a-2a640d2c9d72\") " pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-799fnb2vq" Nov 01 20:18:14.291126 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:14.291048 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-799fnb2vq" Nov 01 20:18:14.600033 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:14.600012 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-799fnb2vq"] Nov 01 20:18:14.601191 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:18:14.601158 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8540f99e_96ca_45f5_922a_2a640d2c9d72.slice/crio-bbdd7193a1ff2d7aed7d9b0a2e15a99639d4062e3d78799556d35413289d5e7c WatchSource:0}: Error finding container bbdd7193a1ff2d7aed7d9b0a2e15a99639d4062e3d78799556d35413289d5e7c: Status 404 returned error can't find the container with id bbdd7193a1ff2d7aed7d9b0a2e15a99639d4062e3d78799556d35413289d5e7c Nov 01 20:18:15.202152 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:15.202112 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-799fnb2vq" event={"ID":"8540f99e-96ca-45f5-922a-2a640d2c9d72","Type":"ContainerStarted","Data":"00f13d2bd84bfe82a8a74f5969cd780305ef7025ffe92e427ec3c2b6e32e2765"} Nov 01 20:18:15.202673 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:15.202179 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-799fnb2vq" event={"ID":"8540f99e-96ca-45f5-922a-2a640d2c9d72","Type":"ContainerStarted","Data":"bbdd7193a1ff2d7aed7d9b0a2e15a99639d4062e3d78799556d35413289d5e7c"} Nov 01 20:18:15.202673 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:15.202201 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-799fnb2vq" Nov 01 20:18:15.219467 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:15.219412 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-799fnb2vq" podStartSLOduration=2.219395973 podStartE2EDuration="2.219395973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:18:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:18:15.21758812 +0000 UTC m=+1140.990748632" watchObservedRunningTime="2026-11-01 20:18:15.219395973 +0000 UTC m=+1140.992556486" Nov 01 20:18:22.204310 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:22.196445 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-68f76b7b94-ghwhj" Nov 01 20:18:26.211072 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:26.211038 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-799fnb2vq" Nov 01 20:18:35.845111 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:18:35.845074 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:18:35.973040 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:35.972954 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-799fnb2vq"] Nov 01 20:18:40.974424 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:40.974350 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-799fnb2vq" podUID="8540f99e-96ca-45f5-922a-2a640d2c9d72" containerName="payload-processing" containerID="cri-o://00f13d2bd84bfe82a8a74f5969cd780305ef7025ffe92e427ec3c2b6e32e2765" gracePeriod=25 Nov 01 20:18:41.259736 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:18:41.259647 5935 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8540f99e_96ca_45f5_922a_2a640d2c9d72.slice/crio-00f13d2bd84bfe82a8a74f5969cd780305ef7025ffe92e427ec3c2b6e32e2765.scope\": RecentStats: unable to find data in memory cache]" Nov 01 20:18:41.620023 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:41.619815 5935 generic.go:356] "Generic (PLEG): container finished" podID="8540f99e-96ca-45f5-922a-2a640d2c9d72" containerID="00f13d2bd84bfe82a8a74f5969cd780305ef7025ffe92e427ec3c2b6e32e2765" exitCode=0 Nov 01 20:18:41.620023 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:41.619913 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-799fnb2vq" event={"ID":"8540f99e-96ca-45f5-922a-2a640d2c9d72","Type":"ContainerDied","Data":"00f13d2bd84bfe82a8a74f5969cd780305ef7025ffe92e427ec3c2b6e32e2765"} Nov 01 20:18:41.620023 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:41.619964 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-799fnb2vq" event={"ID":"8540f99e-96ca-45f5-922a-2a640d2c9d72","Type":"ContainerDied","Data":"bbdd7193a1ff2d7aed7d9b0a2e15a99639d4062e3d78799556d35413289d5e7c"} Nov 01 20:18:41.620023 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:41.619988 5935 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bbdd7193a1ff2d7aed7d9b0a2e15a99639d4062e3d78799556d35413289d5e7c" Nov 01 20:18:41.630732 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:41.630702 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-799fnb2vq" Nov 01 20:18:41.638931 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:41.638907 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8540f99e-96ca-45f5-922a-2a640d2c9d72-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8540f99e-96ca-45f5-922a-2a640d2c9d72-provider-credentials\") pod \"8540f99e-96ca-45f5-922a-2a640d2c9d72\" (UID: \"8540f99e-96ca-45f5-922a-2a640d2c9d72\") " Nov 01 20:18:41.639135 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:41.639123 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8540f99e-96ca-45f5-922a-2a640d2c9d72-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8540f99e-96ca-45f5-922a-2a640d2c9d72-routing-overlay-volume\") pod \"8540f99e-96ca-45f5-922a-2a640d2c9d72\" (UID: \"8540f99e-96ca-45f5-922a-2a640d2c9d72\") " Nov 01 20:18:41.639184 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:41.639147 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8540f99e-96ca-45f5-922a-2a640d2c9d72-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8540f99e-96ca-45f5-922a-2a640d2c9d72-plugins-config-volume\") pod \"8540f99e-96ca-45f5-922a-2a640d2c9d72\" (UID: \"8540f99e-96ca-45f5-922a-2a640d2c9d72\") " Nov 01 20:18:41.639227 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:41.639183 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8540f99e-96ca-45f5-922a-2a640d2c9d72-kube-api-access-788ph\" (UniqueName: \"kubernetes.io/projected/8540f99e-96ca-45f5-922a-2a640d2c9d72-kube-api-access-788ph\") pod \"8540f99e-96ca-45f5-922a-2a640d2c9d72\" (UID: \"8540f99e-96ca-45f5-922a-2a640d2c9d72\") " Nov 01 20:18:41.640093 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:41.640069 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8540f99e-96ca-45f5-922a-2a640d2c9d72-routing-overlay-volume" pod "8540f99e-96ca-45f5-922a-2a640d2c9d72" (UID: "8540f99e-96ca-45f5-922a-2a640d2c9d72"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:18:41.640201 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:41.640098 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8540f99e-96ca-45f5-922a-2a640d2c9d72-plugins-config-volume" pod "8540f99e-96ca-45f5-922a-2a640d2c9d72" (UID: "8540f99e-96ca-45f5-922a-2a640d2c9d72"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:18:41.641582 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:41.641554 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8540f99e-96ca-45f5-922a-2a640d2c9d72-kube-api-access-788ph" pod "8540f99e-96ca-45f5-922a-2a640d2c9d72" (UID: "8540f99e-96ca-45f5-922a-2a640d2c9d72"). InnerVolumeSpecName "kube-api-access-788ph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:18:41.641758 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:41.641739 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8540f99e-96ca-45f5-922a-2a640d2c9d72-provider-credentials" pod "8540f99e-96ca-45f5-922a-2a640d2c9d72" (UID: "8540f99e-96ca-45f5-922a-2a640d2c9d72"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:18:41.740277 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:41.740235 5935 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8540f99e-96ca-45f5-922a-2a640d2c9d72-routing-overlay-volume\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:18:41.740277 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:41.740265 5935 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8540f99e-96ca-45f5-922a-2a640d2c9d72-plugins-config-volume\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:18:41.740277 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:41.740274 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-788ph\" (UniqueName: \"kubernetes.io/projected/8540f99e-96ca-45f5-922a-2a640d2c9d72-kube-api-access-788ph\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:18:41.740277 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:41.740312 5935 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8540f99e-96ca-45f5-922a-2a640d2c9d72-provider-credentials\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:18:41.790017 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:41.789905 5935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-ljtcq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 20:18:41.790337 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:41.790275 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-ljtcq" podUID="03014616-4520-48bb-a548-66485577e26c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:18:42.626960 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:42.626876 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-799fnb2vq" Nov 01 20:18:42.643619 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:42.643585 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-799fnb2vq"] Nov 01 20:18:42.648811 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:42.648788 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-29b57c/payload-processing-external-model-e2e-extmod-a-29b57c-799fnb2vq"] Nov 01 20:18:44.356807 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:44.356772 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8540f99e-96ca-45f5-922a-2a640d2c9d72" path="/var/lib/kubelet/pods/8540f99e-96ca-45f5-922a-2a640d2c9d72/volumes" Nov 01 20:18:56.768472 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:56.760764 5935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-ljtcq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 20:18:56.768472 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:56.760838 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-ljtcq" podUID="03014616-4520-48bb-a548-66485577e26c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:18:57.092447 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:18:57.092360 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-68f76b7b94-ghwhj"] Nov 01 20:19:02.093935 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:19:02.093890 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-68f76b7b94-ghwhj" podUID="c293efa4-25da-468d-8f4e-12ebd296a217" containerName="payload-processing" containerID="cri-o://9546eee773590057ead2ed95e2bc095de5767800e713ca0477366e84566432cd" gracePeriod=25 Nov 01 20:19:02.179873 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:19:02.179836 5935 logging.go:55] [core] [Channel #1163 SubChannel #1164]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.87:50052", ServerName: "10.133.0.87:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.87:50052: connect: connection refused" Nov 01 20:19:02.381475 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:19:02.381453 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-68f76b7b94-ghwhj" Nov 01 20:19:02.473716 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:19:02.473687 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c293efa4-25da-468d-8f4e-12ebd296a217-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c293efa4-25da-468d-8f4e-12ebd296a217-plugins-config-volume\") pod \"c293efa4-25da-468d-8f4e-12ebd296a217\" (UID: \"c293efa4-25da-468d-8f4e-12ebd296a217\") " Nov 01 20:19:02.473877 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:19:02.473722 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c293efa4-25da-468d-8f4e-12ebd296a217-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c293efa4-25da-468d-8f4e-12ebd296a217-routing-overlay-volume\") pod \"c293efa4-25da-468d-8f4e-12ebd296a217\" (UID: \"c293efa4-25da-468d-8f4e-12ebd296a217\") " Nov 01 20:19:02.473937 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:19:02.473890 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c293efa4-25da-468d-8f4e-12ebd296a217-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c293efa4-25da-468d-8f4e-12ebd296a217-provider-credentials\") pod \"c293efa4-25da-468d-8f4e-12ebd296a217\" (UID: \"c293efa4-25da-468d-8f4e-12ebd296a217\") " Nov 01 20:19:02.473937 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:19:02.473925 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c293efa4-25da-468d-8f4e-12ebd296a217-kube-api-access-rmjhg\" (UniqueName: \"kubernetes.io/projected/c293efa4-25da-468d-8f4e-12ebd296a217-kube-api-access-rmjhg\") pod \"c293efa4-25da-468d-8f4e-12ebd296a217\" (UID: \"c293efa4-25da-468d-8f4e-12ebd296a217\") " Nov 01 20:19:02.474080 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:19:02.474058 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c293efa4-25da-468d-8f4e-12ebd296a217-plugins-config-volume" pod "c293efa4-25da-468d-8f4e-12ebd296a217" (UID: "c293efa4-25da-468d-8f4e-12ebd296a217"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:19:02.474176 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:19:02.474150 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c293efa4-25da-468d-8f4e-12ebd296a217-routing-overlay-volume" pod "c293efa4-25da-468d-8f4e-12ebd296a217" (UID: "c293efa4-25da-468d-8f4e-12ebd296a217"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:19:02.474474 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:19:02.474453 5935 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c293efa4-25da-468d-8f4e-12ebd296a217-plugins-config-volume\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:19:02.474532 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:19:02.474480 5935 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c293efa4-25da-468d-8f4e-12ebd296a217-routing-overlay-volume\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:19:02.476139 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:19:02.476115 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c293efa4-25da-468d-8f4e-12ebd296a217-kube-api-access-rmjhg" pod "c293efa4-25da-468d-8f4e-12ebd296a217" (UID: "c293efa4-25da-468d-8f4e-12ebd296a217"). InnerVolumeSpecName "kube-api-access-rmjhg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:19:02.476215 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:19:02.476135 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c293efa4-25da-468d-8f4e-12ebd296a217-provider-credentials" pod "c293efa4-25da-468d-8f4e-12ebd296a217" (UID: "c293efa4-25da-468d-8f4e-12ebd296a217"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:19:02.576015 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:19:02.575985 5935 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c293efa4-25da-468d-8f4e-12ebd296a217-provider-credentials\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:19:02.576015 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:19:02.576012 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmjhg\" (UniqueName: \"kubernetes.io/projected/c293efa4-25da-468d-8f4e-12ebd296a217-kube-api-access-rmjhg\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:19:03.033499 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:19:03.033464 5935 generic.go:356] "Generic (PLEG): container finished" podID="c293efa4-25da-468d-8f4e-12ebd296a217" containerID="9546eee773590057ead2ed95e2bc095de5767800e713ca0477366e84566432cd" exitCode=0 Nov 01 20:19:03.033683 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:19:03.033509 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-68f76b7b94-ghwhj" Nov 01 20:19:03.033683 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:19:03.033551 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-68f76b7b94-ghwhj" event={"ID":"c293efa4-25da-468d-8f4e-12ebd296a217","Type":"ContainerDied","Data":"9546eee773590057ead2ed95e2bc095de5767800e713ca0477366e84566432cd"} Nov 01 20:19:03.033683 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:19:03.033609 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-68f76b7b94-ghwhj" event={"ID":"c293efa4-25da-468d-8f4e-12ebd296a217","Type":"ContainerDied","Data":"2152093139c88e75cd5458bbac3c5e291653caa06028901e68c3b1f695ceba1e"} Nov 01 20:19:03.033683 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:19:03.033636 5935 scope.go:122] "RemoveContainer" containerID="9546eee773590057ead2ed95e2bc095de5767800e713ca0477366e84566432cd" Nov 01 20:19:03.045505 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:19:03.045482 5935 scope.go:122] "RemoveContainer" containerID="9546eee773590057ead2ed95e2bc095de5767800e713ca0477366e84566432cd" Nov 01 20:19:03.045962 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:19:03.045925 5935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9546eee773590057ead2ed95e2bc095de5767800e713ca0477366e84566432cd\": container with ID starting with 9546eee773590057ead2ed95e2bc095de5767800e713ca0477366e84566432cd not found: ID does not exist" containerID="9546eee773590057ead2ed95e2bc095de5767800e713ca0477366e84566432cd" Nov 01 20:19:03.046076 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:19:03.045967 5935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9546eee773590057ead2ed95e2bc095de5767800e713ca0477366e84566432cd"} err="failed to get container status \"9546eee773590057ead2ed95e2bc095de5767800e713ca0477366e84566432cd\": rpc error: code = NotFound desc = could not find container \"9546eee773590057ead2ed95e2bc095de5767800e713ca0477366e84566432cd\": container with ID starting with 9546eee773590057ead2ed95e2bc095de5767800e713ca0477366e84566432cd not found: ID does not exist" Nov 01 20:19:03.057145 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:19:03.057072 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-68f76b7b94-ghwhj"] Nov 01 20:19:03.060129 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:19:03.060108 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-68f76b7b94-ghwhj"] Nov 01 20:19:03.180115 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:19:03.180079 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-68f76b7b94-ghwhj" podUID="c293efa4-25da-468d-8f4e-12ebd296a217" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.87:50052\" within 1s: context deadline exceeded" Nov 01 20:19:04.356936 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:19:04.356903 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c293efa4-25da-468d-8f4e-12ebd296a217" path="/var/lib/kubelet/pods/c293efa4-25da-468d-8f4e-12ebd296a217/volumes" Nov 01 20:19:05.844465 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:19:05.844426 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:19:18.633379 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:19:18.632404 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6afe1a-5f58b8d4d5-s7lmk"] Nov 01 20:19:18.634850 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:19:18.634560 5935 cpu_manager.go:414] "containerMap: removing container" podUID="c293efa4-25da-468d-8f4e-12ebd296a217" containerName="payload-processing" Nov 01 20:19:18.634850 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:19:18.634583 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c293efa4-25da-468d-8f4e-12ebd296a217" containerName="payload-processing" Nov 01 20:19:18.634850 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:19:18.634605 5935 cpu_manager.go:414] "containerMap: removing container" podUID="8540f99e-96ca-45f5-922a-2a640d2c9d72" containerName="payload-processing" Nov 01 20:19:18.634850 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:19:18.634612 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8540f99e-96ca-45f5-922a-2a640d2c9d72" containerName="payload-processing" Nov 01 20:19:18.636933 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:19:18.634767 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="c293efa4-25da-468d-8f4e-12ebd296a217" containerName="payload-processing" Nov 01 20:19:18.636933 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:19:18.635141 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="8540f99e-96ca-45f5-922a-2a640d2c9d72" containerName="payload-processing" Nov 01 20:19:18.641901 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:19:18.640918 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6afe1a-5f58b8d4d5-s7lmk" Nov 01 20:19:18.657887 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:19:18.650112 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-6afe1a\"" Nov 01 20:19:18.657887 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:19:18.654681 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6afe1a-5f58b8d4d5-s7lmk"] Nov 01 20:19:18.657887 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:19:18.655952 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-6afe1a-dockercfg-6qh7v\"" Nov 01 20:19:18.867308 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:19:18.863395 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bf774f2-2df3-42f7-b64b-a97186e73899-plugins-config-volume\") pod \"payload-processing-e2e-api-b-6afe1a-5f58b8d4d5-s7lmk\" (UID: \"7bf774f2-2df3-42f7-b64b-a97186e73899\") " pod="openshift-ingress/payload-processing-e2e-api-b-6afe1a-5f58b8d4d5-s7lmk" Nov 01 20:19:18.867308 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:19:18.863683 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdzsz\" (UniqueName: \"kubernetes.io/projected/7bf774f2-2df3-42f7-b64b-a97186e73899-kube-api-access-cdzsz\") pod \"payload-processing-e2e-api-b-6afe1a-5f58b8d4d5-s7lmk\" (UID: \"7bf774f2-2df3-42f7-b64b-a97186e73899\") " pod="openshift-ingress/payload-processing-e2e-api-b-6afe1a-5f58b8d4d5-s7lmk" Nov 01 20:19:18.999697 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:19:18.999662 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bf774f2-2df3-42f7-b64b-a97186e73899-plugins-config-volume\") pod \"payload-processing-e2e-api-b-6afe1a-5f58b8d4d5-s7lmk\" (UID: \"7bf774f2-2df3-42f7-b64b-a97186e73899\") " pod="openshift-ingress/payload-processing-e2e-api-b-6afe1a-5f58b8d4d5-s7lmk" Nov 01 20:19:19.001274 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:19:19.001249 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdzsz\" (UniqueName: \"kubernetes.io/projected/7bf774f2-2df3-42f7-b64b-a97186e73899-kube-api-access-cdzsz\") pod \"payload-processing-e2e-api-b-6afe1a-5f58b8d4d5-s7lmk\" (UID: \"7bf774f2-2df3-42f7-b64b-a97186e73899\") " pod="openshift-ingress/payload-processing-e2e-api-b-6afe1a-5f58b8d4d5-s7lmk" Nov 01 20:19:19.001862 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:19:19.000931 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bf774f2-2df3-42f7-b64b-a97186e73899-plugins-config-volume\") pod \"payload-processing-e2e-api-b-6afe1a-5f58b8d4d5-s7lmk\" (UID: \"7bf774f2-2df3-42f7-b64b-a97186e73899\") " pod="openshift-ingress/payload-processing-e2e-api-b-6afe1a-5f58b8d4d5-s7lmk" Nov 01 20:19:19.081552 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:19:19.081325 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdzsz\" (UniqueName: \"kubernetes.io/projected/7bf774f2-2df3-42f7-b64b-a97186e73899-kube-api-access-cdzsz\") pod \"payload-processing-e2e-api-b-6afe1a-5f58b8d4d5-s7lmk\" (UID: \"7bf774f2-2df3-42f7-b64b-a97186e73899\") " pod="openshift-ingress/payload-processing-e2e-api-b-6afe1a-5f58b8d4d5-s7lmk" Nov 01 20:19:19.365532 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:19:19.365443 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6afe1a-5f58b8d4d5-s7lmk" Nov 01 20:19:19.899887 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:19:19.839551 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6afe1a-5f58b8d4d5-s7lmk"] Nov 01 20:19:20.229559 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:19:20.229521 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6afe1a-5f58b8d4d5-s7lmk" event={"ID":"7bf774f2-2df3-42f7-b64b-a97186e73899","Type":"ContainerStarted","Data":"c3ae63fd7bda5c7c669362a5d0a162484b53e32b8c8eaf4215af12f0cba6e2ad"} Nov 01 20:19:20.229711 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:19:20.229577 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6afe1a-5f58b8d4d5-s7lmk" event={"ID":"7bf774f2-2df3-42f7-b64b-a97186e73899","Type":"ContainerStarted","Data":"aa599cf8473d762b02bd8d9d9749efe9f3dc29f6a3d2fd14b04205c0cb370123"} Nov 01 20:19:20.229780 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:19:20.229765 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-6afe1a-5f58b8d4d5-s7lmk" Nov 01 20:19:31.238038 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:19:31.238003 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-6afe1a-5f58b8d4d5-s7lmk" Nov 01 20:19:31.254061 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:19:31.254010 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-6afe1a-5f58b8d4d5-s7lmk" podStartSLOduration=13.253997162 podStartE2EDuration="13.253997162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:19:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:19:20.248941487 +0000 UTC m=+1206.022102000" watchObservedRunningTime="2026-11-01 20:19:31.253997162 +0000 UTC m=+1217.027157674" Nov 01 20:19:35.844899 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:19:35.844863 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:19:41.719850 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:19:41.719399 5935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-ljtcq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 20:19:41.719850 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:19:41.719461 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-ljtcq" podUID="03014616-4520-48bb-a548-66485577e26c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:20:04.749729 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:04.749690 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6afe1a-5f58b8d4d5-s7lmk"] Nov 01 20:20:05.844817 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:20:05.844782 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:20:09.750474 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:09.750429 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-6afe1a-5f58b8d4d5-s7lmk" podUID="7bf774f2-2df3-42f7-b64b-a97186e73899" containerName="payload-processing" containerID="cri-o://c3ae63fd7bda5c7c669362a5d0a162484b53e32b8c8eaf4215af12f0cba6e2ad" gracePeriod=25 Nov 01 20:20:10.033742 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:10.033713 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6afe1a-5f58b8d4d5-s7lmk" Nov 01 20:20:10.120342 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:10.120275 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bf774f2-2df3-42f7-b64b-a97186e73899-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bf774f2-2df3-42f7-b64b-a97186e73899-plugins-config-volume\") pod \"7bf774f2-2df3-42f7-b64b-a97186e73899\" (UID: \"7bf774f2-2df3-42f7-b64b-a97186e73899\") " Nov 01 20:20:10.120549 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:10.120491 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bf774f2-2df3-42f7-b64b-a97186e73899-kube-api-access-cdzsz\" (UniqueName: \"kubernetes.io/projected/7bf774f2-2df3-42f7-b64b-a97186e73899-kube-api-access-cdzsz\") pod \"7bf774f2-2df3-42f7-b64b-a97186e73899\" (UID: \"7bf774f2-2df3-42f7-b64b-a97186e73899\") " Nov 01 20:20:10.120662 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:10.120638 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bf774f2-2df3-42f7-b64b-a97186e73899-plugins-config-volume" pod "7bf774f2-2df3-42f7-b64b-a97186e73899" (UID: "7bf774f2-2df3-42f7-b64b-a97186e73899"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:20:10.120999 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:10.120980 5935 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bf774f2-2df3-42f7-b64b-a97186e73899-plugins-config-volume\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:20:10.122560 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:10.122534 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bf774f2-2df3-42f7-b64b-a97186e73899-kube-api-access-cdzsz" pod "7bf774f2-2df3-42f7-b64b-a97186e73899" (UID: "7bf774f2-2df3-42f7-b64b-a97186e73899"). InnerVolumeSpecName "kube-api-access-cdzsz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:20:10.222023 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:10.221989 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdzsz\" (UniqueName: \"kubernetes.io/projected/7bf774f2-2df3-42f7-b64b-a97186e73899-kube-api-access-cdzsz\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:20:10.619433 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:10.619322 5935 generic.go:356] "Generic (PLEG): container finished" podID="7bf774f2-2df3-42f7-b64b-a97186e73899" containerID="c3ae63fd7bda5c7c669362a5d0a162484b53e32b8c8eaf4215af12f0cba6e2ad" exitCode=0 Nov 01 20:20:10.619433 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:10.619390 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-6afe1a-5f58b8d4d5-s7lmk" Nov 01 20:20:10.619674 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:10.619444 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6afe1a-5f58b8d4d5-s7lmk" event={"ID":"7bf774f2-2df3-42f7-b64b-a97186e73899","Type":"ContainerDied","Data":"c3ae63fd7bda5c7c669362a5d0a162484b53e32b8c8eaf4215af12f0cba6e2ad"} Nov 01 20:20:10.619674 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:10.619508 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-6afe1a-5f58b8d4d5-s7lmk" event={"ID":"7bf774f2-2df3-42f7-b64b-a97186e73899","Type":"ContainerDied","Data":"aa599cf8473d762b02bd8d9d9749efe9f3dc29f6a3d2fd14b04205c0cb370123"} Nov 01 20:20:10.619674 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:10.619547 5935 scope.go:122] "RemoveContainer" containerID="c3ae63fd7bda5c7c669362a5d0a162484b53e32b8c8eaf4215af12f0cba6e2ad" Nov 01 20:20:10.633794 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:10.633776 5935 scope.go:122] "RemoveContainer" containerID="c3ae63fd7bda5c7c669362a5d0a162484b53e32b8c8eaf4215af12f0cba6e2ad" Nov 01 20:20:10.634121 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:20:10.634100 5935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3ae63fd7bda5c7c669362a5d0a162484b53e32b8c8eaf4215af12f0cba6e2ad\": container with ID starting with c3ae63fd7bda5c7c669362a5d0a162484b53e32b8c8eaf4215af12f0cba6e2ad not found: ID does not exist" containerID="c3ae63fd7bda5c7c669362a5d0a162484b53e32b8c8eaf4215af12f0cba6e2ad" Nov 01 20:20:10.634198 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:10.634132 5935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3ae63fd7bda5c7c669362a5d0a162484b53e32b8c8eaf4215af12f0cba6e2ad"} err="failed to get container status \"c3ae63fd7bda5c7c669362a5d0a162484b53e32b8c8eaf4215af12f0cba6e2ad\": rpc error: code = NotFound desc = could not find container \"c3ae63fd7bda5c7c669362a5d0a162484b53e32b8c8eaf4215af12f0cba6e2ad\": container with ID starting with c3ae63fd7bda5c7c669362a5d0a162484b53e32b8c8eaf4215af12f0cba6e2ad not found: ID does not exist" Nov 01 20:20:10.640634 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:10.640610 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6afe1a-5f58b8d4d5-s7lmk"] Nov 01 20:20:10.643964 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:10.643938 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-6afe1a-5f58b8d4d5-s7lmk"] Nov 01 20:20:12.353758 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:12.353727 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bf774f2-2df3-42f7-b64b-a97186e73899" path="/var/lib/kubelet/pods/7bf774f2-2df3-42f7-b64b-a97186e73899/volumes" Nov 01 20:20:12.570414 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:12.570380 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-9be196/auth-test-model-9be196-kserve-587468665c-b2xdw"] Nov 01 20:20:24.334543 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:24.334511 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-a6873b-79b7f97879-9dhts"] Nov 01 20:20:24.335272 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:24.335250 5935 cpu_manager.go:414] "containerMap: removing container" podUID="7bf774f2-2df3-42f7-b64b-a97186e73899" containerName="payload-processing" Nov 01 20:20:24.335390 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:24.335273 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bf774f2-2df3-42f7-b64b-a97186e73899" containerName="payload-processing" Nov 01 20:20:24.335494 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:24.335480 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bf774f2-2df3-42f7-b64b-a97186e73899" containerName="payload-processing" Nov 01 20:20:24.341349 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:24.341325 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-a6873b-79b7f97879-9dhts" Nov 01 20:20:24.359585 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:24.359557 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-a6873b-79b7f97879-9dhts"] Nov 01 20:20:24.408676 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:24.408629 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9zfv\" (UniqueName: \"kubernetes.io/projected/d20477dc-2e11-4469-879d-eb1629c24898-kube-api-access-n9zfv\") pod \"payload-processing-e2e-worker-w1-a6873b-79b7f97879-9dhts\" (UID: \"d20477dc-2e11-4469-879d-eb1629c24898\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-a6873b-79b7f97879-9dhts" Nov 01 20:20:24.408919 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:24.408738 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d20477dc-2e11-4469-879d-eb1629c24898-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-a6873b-79b7f97879-9dhts\" (UID: \"d20477dc-2e11-4469-879d-eb1629c24898\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-a6873b-79b7f97879-9dhts" Nov 01 20:20:24.432617 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:24.432581 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-a6873b-dockercfg-sjfqm\"" Nov 01 20:20:24.432824 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:24.432690 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-a6873b\"" Nov 01 20:20:24.510184 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:24.510144 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9zfv\" (UniqueName: \"kubernetes.io/projected/d20477dc-2e11-4469-879d-eb1629c24898-kube-api-access-n9zfv\") pod \"payload-processing-e2e-worker-w1-a6873b-79b7f97879-9dhts\" (UID: \"d20477dc-2e11-4469-879d-eb1629c24898\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-a6873b-79b7f97879-9dhts" Nov 01 20:20:24.510626 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:24.510572 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d20477dc-2e11-4469-879d-eb1629c24898-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-a6873b-79b7f97879-9dhts\" (UID: \"d20477dc-2e11-4469-879d-eb1629c24898\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-a6873b-79b7f97879-9dhts" Nov 01 20:20:24.521392 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:24.511894 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d20477dc-2e11-4469-879d-eb1629c24898-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-a6873b-79b7f97879-9dhts\" (UID: \"d20477dc-2e11-4469-879d-eb1629c24898\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-a6873b-79b7f97879-9dhts" Nov 01 20:20:24.554252 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:24.554220 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9zfv\" (UniqueName: \"kubernetes.io/projected/d20477dc-2e11-4469-879d-eb1629c24898-kube-api-access-n9zfv\") pod \"payload-processing-e2e-worker-w1-a6873b-79b7f97879-9dhts\" (UID: \"d20477dc-2e11-4469-879d-eb1629c24898\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-a6873b-79b7f97879-9dhts" Nov 01 20:20:24.668024 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:24.667943 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-a6873b-79b7f97879-9dhts" Nov 01 20:20:24.956093 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:24.956063 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-a6873b-79b7f97879-9dhts"] Nov 01 20:20:24.957458 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:20:24.957430 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd20477dc_2e11_4469_879d_eb1629c24898.slice/crio-e7d95fbf2ac02e096706c13fa0511752e086e8f750e100bd927d2aa236167cae WatchSource:0}: Error finding container e7d95fbf2ac02e096706c13fa0511752e086e8f750e100bd927d2aa236167cae: Status 404 returned error can't find the container with id e7d95fbf2ac02e096706c13fa0511752e086e8f750e100bd927d2aa236167cae Nov 01 20:20:25.786766 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:25.786736 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-a6873b-79b7f97879-9dhts" event={"ID":"d20477dc-2e11-4469-879d-eb1629c24898","Type":"ContainerStarted","Data":"8e27bcd5732252c3ce733f5c6b01c9aa5989465f0c7eb4b8ef8ff77685711bcc"} Nov 01 20:20:25.787179 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:25.786788 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-a6873b-79b7f97879-9dhts" event={"ID":"d20477dc-2e11-4469-879d-eb1629c24898","Type":"ContainerStarted","Data":"e7d95fbf2ac02e096706c13fa0511752e086e8f750e100bd927d2aa236167cae"} Nov 01 20:20:25.787179 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:25.786891 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-a6873b-79b7f97879-9dhts" Nov 01 20:20:25.807601 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:25.807554 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-a6873b-79b7f97879-9dhts" podStartSLOduration=1.807542607 podStartE2EDuration="1.807542607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:20:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:20:25.804029002 +0000 UTC m=+1271.577189539" watchObservedRunningTime="2026-11-01 20:20:25.807542607 +0000 UTC m=+1271.580703119" Nov 01 20:20:27.640520 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:27.640473 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-9be196/auth-test-model-9be196-kserve-587468665c-b2xdw" podUID="ffc910ff-b218-4ea3-893e-987a9503a634" containerName="main" containerID="cri-o://790244db417be08a390c18e63999b627438522bacd86bf36805083fb1031460e" gracePeriod=45 Nov 01 20:20:27.775674 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:27.774788 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w1-9be196/auth-test-model-9be196-kserve-587468665c-b2xdw" podUID="ffc910ff-b218-4ea3-893e-987a9503a634" containerName="main" probeResult="failure" output="Get \"https://10.133.0.83:8000/ready\": dial tcp 10.133.0.83:8000: connect: connection refused" Nov 01 20:20:28.519435 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:28.519400 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-9be196/auth-test-model-9be196-kserve-587468665c-b2xdw" Nov 01 20:20:28.547324 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:28.546721 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffc910ff-b218-4ea3-893e-987a9503a634-home\" (UniqueName: \"kubernetes.io/empty-dir/ffc910ff-b218-4ea3-893e-987a9503a634-home\") pod \"ffc910ff-b218-4ea3-893e-987a9503a634\" (UID: \"ffc910ff-b218-4ea3-893e-987a9503a634\") " Nov 01 20:20:28.547324 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:28.546775 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffc910ff-b218-4ea3-893e-987a9503a634-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffc910ff-b218-4ea3-893e-987a9503a634-tmp-dir\") pod \"ffc910ff-b218-4ea3-893e-987a9503a634\" (UID: \"ffc910ff-b218-4ea3-893e-987a9503a634\") " Nov 01 20:20:28.547324 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:28.546938 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffc910ff-b218-4ea3-893e-987a9503a634-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffc910ff-b218-4ea3-893e-987a9503a634-dshm\") pod \"ffc910ff-b218-4ea3-893e-987a9503a634\" (UID: \"ffc910ff-b218-4ea3-893e-987a9503a634\") " Nov 01 20:20:28.547324 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:28.547052 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffc910ff-b218-4ea3-893e-987a9503a634-kube-api-access-z667l\" (UniqueName: \"kubernetes.io/projected/ffc910ff-b218-4ea3-893e-987a9503a634-kube-api-access-z667l\") pod \"ffc910ff-b218-4ea3-893e-987a9503a634\" (UID: \"ffc910ff-b218-4ea3-893e-987a9503a634\") " Nov 01 20:20:28.547324 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:28.547089 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffc910ff-b218-4ea3-893e-987a9503a634-tls-certs\" (UniqueName: \"kubernetes.io/secret/ffc910ff-b218-4ea3-893e-987a9503a634-tls-certs\") pod \"ffc910ff-b218-4ea3-893e-987a9503a634\" (UID: \"ffc910ff-b218-4ea3-893e-987a9503a634\") " Nov 01 20:20:28.547324 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:28.547139 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffc910ff-b218-4ea3-893e-987a9503a634-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffc910ff-b218-4ea3-893e-987a9503a634-model-cache\") pod \"ffc910ff-b218-4ea3-893e-987a9503a634\" (UID: \"ffc910ff-b218-4ea3-893e-987a9503a634\") " Nov 01 20:20:28.577312 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:28.574986 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffc910ff-b218-4ea3-893e-987a9503a634-home" pod "ffc910ff-b218-4ea3-893e-987a9503a634" (UID: "ffc910ff-b218-4ea3-893e-987a9503a634"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:20:28.577312 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:28.576872 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffc910ff-b218-4ea3-893e-987a9503a634-tmp-dir" pod "ffc910ff-b218-4ea3-893e-987a9503a634" (UID: "ffc910ff-b218-4ea3-893e-987a9503a634"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:20:28.613878 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:28.589365 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffc910ff-b218-4ea3-893e-987a9503a634-model-cache" pod "ffc910ff-b218-4ea3-893e-987a9503a634" (UID: "ffc910ff-b218-4ea3-893e-987a9503a634"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:20:28.615338 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:28.615246 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffc910ff-b218-4ea3-893e-987a9503a634-tls-certs" pod "ffc910ff-b218-4ea3-893e-987a9503a634" (UID: "ffc910ff-b218-4ea3-893e-987a9503a634"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:20:28.620552 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:28.620450 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffc910ff-b218-4ea3-893e-987a9503a634-kube-api-access-z667l" pod "ffc910ff-b218-4ea3-893e-987a9503a634" (UID: "ffc910ff-b218-4ea3-893e-987a9503a634"). InnerVolumeSpecName "kube-api-access-z667l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:20:28.620552 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:28.620523 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffc910ff-b218-4ea3-893e-987a9503a634-dshm" pod "ffc910ff-b218-4ea3-893e-987a9503a634" (UID: "ffc910ff-b218-4ea3-893e-987a9503a634"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:20:28.650469 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:28.650329 5935 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffc910ff-b218-4ea3-893e-987a9503a634-dshm\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:20:28.650469 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:28.650360 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z667l\" (UniqueName: \"kubernetes.io/projected/ffc910ff-b218-4ea3-893e-987a9503a634-kube-api-access-z667l\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:20:28.650469 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:28.650377 5935 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffc910ff-b218-4ea3-893e-987a9503a634-tls-certs\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:20:28.650469 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:28.650392 5935 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffc910ff-b218-4ea3-893e-987a9503a634-model-cache\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:20:28.650469 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:28.650407 5935 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffc910ff-b218-4ea3-893e-987a9503a634-home\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:20:28.650469 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:28.650447 5935 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffc910ff-b218-4ea3-893e-987a9503a634-tmp-dir\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:20:28.939722 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:28.939692 5935 generic.go:356] "Generic (PLEG): container finished" podID="ffc910ff-b218-4ea3-893e-987a9503a634" containerID="790244db417be08a390c18e63999b627438522bacd86bf36805083fb1031460e" exitCode=0 Nov 01 20:20:28.939906 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:28.939730 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-9be196/auth-test-model-9be196-kserve-587468665c-b2xdw" Nov 01 20:20:28.939906 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:28.939775 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-9be196/auth-test-model-9be196-kserve-587468665c-b2xdw" event={"ID":"ffc910ff-b218-4ea3-893e-987a9503a634","Type":"ContainerDied","Data":"790244db417be08a390c18e63999b627438522bacd86bf36805083fb1031460e"} Nov 01 20:20:28.939906 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:28.939821 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-9be196/auth-test-model-9be196-kserve-587468665c-b2xdw" event={"ID":"ffc910ff-b218-4ea3-893e-987a9503a634","Type":"ContainerDied","Data":"2be413a6355d00bd3a697c4daae6cbccc5290cc1ee06def6de227934ec6f1a5d"} Nov 01 20:20:28.939906 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:28.939843 5935 scope.go:122] "RemoveContainer" containerID="790244db417be08a390c18e63999b627438522bacd86bf36805083fb1031460e" Nov 01 20:20:28.955730 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:28.955257 5935 scope.go:122] "RemoveContainer" containerID="790244db417be08a390c18e63999b627438522bacd86bf36805083fb1031460e" Nov 01 20:20:28.955730 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:20:28.955634 5935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"790244db417be08a390c18e63999b627438522bacd86bf36805083fb1031460e\": container with ID starting with 790244db417be08a390c18e63999b627438522bacd86bf36805083fb1031460e not found: ID does not exist" containerID="790244db417be08a390c18e63999b627438522bacd86bf36805083fb1031460e" Nov 01 20:20:28.955730 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:28.955666 5935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"790244db417be08a390c18e63999b627438522bacd86bf36805083fb1031460e"} err="failed to get container status \"790244db417be08a390c18e63999b627438522bacd86bf36805083fb1031460e\": rpc error: code = NotFound desc = could not find container \"790244db417be08a390c18e63999b627438522bacd86bf36805083fb1031460e\": container with ID starting with 790244db417be08a390c18e63999b627438522bacd86bf36805083fb1031460e not found: ID does not exist" Nov 01 20:20:29.019396 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:29.019335 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-9be196/auth-test-model-9be196-kserve-587468665c-b2xdw"] Nov 01 20:20:29.023851 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:29.023823 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-9be196/auth-test-model-9be196-kserve-587468665c-b2xdw"] Nov 01 20:20:30.354391 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:30.354359 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffc910ff-b218-4ea3-893e-987a9503a634" path="/var/lib/kubelet/pods/ffc910ff-b218-4ea3-893e-987a9503a634/volumes" Nov 01 20:20:33.303324 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:33.301219 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-933245-7cd656f7b9-7lszc"] Nov 01 20:20:33.303324 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:33.302181 5935 cpu_manager.go:414] "containerMap: removing container" podUID="ffc910ff-b218-4ea3-893e-987a9503a634" containerName="main" Nov 01 20:20:33.303324 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:33.302205 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffc910ff-b218-4ea3-893e-987a9503a634" containerName="main" Nov 01 20:20:33.303324 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:33.302414 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffc910ff-b218-4ea3-893e-987a9503a634" containerName="main" Nov 01 20:20:33.309329 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:33.305996 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-933245-7cd656f7b9-7lszc"] Nov 01 20:20:33.309329 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:33.306079 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-933245-7cd656f7b9-7lszc" Nov 01 20:20:33.348660 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:33.348635 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-933245-serving-cert\"" Nov 01 20:20:33.433017 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:33.432987 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgkfj\" (UniqueName: \"kubernetes.io/projected/7f516592-de0d-441d-aa8a-85a22568a94b-kube-api-access-mgkfj\") pod \"maas-api-e2e-autores-933245-7cd656f7b9-7lszc\" (UID: \"7f516592-de0d-441d-aa8a-85a22568a94b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-933245-7cd656f7b9-7lszc" Nov 01 20:20:33.433193 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:33.433023 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f516592-de0d-441d-aa8a-85a22568a94b-openshift-service-ca\") pod \"maas-api-e2e-autores-933245-7cd656f7b9-7lszc\" (UID: \"7f516592-de0d-441d-aa8a-85a22568a94b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-933245-7cd656f7b9-7lszc" Nov 01 20:20:33.433193 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:33.433054 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f516592-de0d-441d-aa8a-85a22568a94b-maas-api-tls\") pod \"maas-api-e2e-autores-933245-7cd656f7b9-7lszc\" (UID: \"7f516592-de0d-441d-aa8a-85a22568a94b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-933245-7cd656f7b9-7lszc" Nov 01 20:20:33.433277 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:33.433195 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f516592-de0d-441d-aa8a-85a22568a94b-metrics-certs\") pod \"maas-api-e2e-autores-933245-7cd656f7b9-7lszc\" (UID: \"7f516592-de0d-441d-aa8a-85a22568a94b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-933245-7cd656f7b9-7lszc" Nov 01 20:20:33.534626 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:33.534589 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgkfj\" (UniqueName: \"kubernetes.io/projected/7f516592-de0d-441d-aa8a-85a22568a94b-kube-api-access-mgkfj\") pod \"maas-api-e2e-autores-933245-7cd656f7b9-7lszc\" (UID: \"7f516592-de0d-441d-aa8a-85a22568a94b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-933245-7cd656f7b9-7lszc" Nov 01 20:20:33.534831 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:33.534645 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f516592-de0d-441d-aa8a-85a22568a94b-openshift-service-ca\") pod \"maas-api-e2e-autores-933245-7cd656f7b9-7lszc\" (UID: \"7f516592-de0d-441d-aa8a-85a22568a94b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-933245-7cd656f7b9-7lszc" Nov 01 20:20:33.534831 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:33.534693 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f516592-de0d-441d-aa8a-85a22568a94b-maas-api-tls\") pod \"maas-api-e2e-autores-933245-7cd656f7b9-7lszc\" (UID: \"7f516592-de0d-441d-aa8a-85a22568a94b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-933245-7cd656f7b9-7lszc" Nov 01 20:20:33.534831 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:33.534737 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f516592-de0d-441d-aa8a-85a22568a94b-metrics-certs\") pod \"maas-api-e2e-autores-933245-7cd656f7b9-7lszc\" (UID: \"7f516592-de0d-441d-aa8a-85a22568a94b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-933245-7cd656f7b9-7lszc" Nov 01 20:20:33.535699 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:33.535669 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f516592-de0d-441d-aa8a-85a22568a94b-openshift-service-ca\") pod \"maas-api-e2e-autores-933245-7cd656f7b9-7lszc\" (UID: \"7f516592-de0d-441d-aa8a-85a22568a94b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-933245-7cd656f7b9-7lszc" Nov 01 20:20:33.538068 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:33.538047 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f516592-de0d-441d-aa8a-85a22568a94b-metrics-certs\") pod \"maas-api-e2e-autores-933245-7cd656f7b9-7lszc\" (UID: \"7f516592-de0d-441d-aa8a-85a22568a94b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-933245-7cd656f7b9-7lszc" Nov 01 20:20:33.538237 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:33.538214 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f516592-de0d-441d-aa8a-85a22568a94b-maas-api-tls\") pod \"maas-api-e2e-autores-933245-7cd656f7b9-7lszc\" (UID: \"7f516592-de0d-441d-aa8a-85a22568a94b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-933245-7cd656f7b9-7lszc" Nov 01 20:20:33.542703 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:33.542683 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgkfj\" (UniqueName: \"kubernetes.io/projected/7f516592-de0d-441d-aa8a-85a22568a94b-kube-api-access-mgkfj\") pod \"maas-api-e2e-autores-933245-7cd656f7b9-7lszc\" (UID: \"7f516592-de0d-441d-aa8a-85a22568a94b\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-933245-7cd656f7b9-7lszc" Nov 01 20:20:33.624085 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:33.624018 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-933245-7cd656f7b9-7lszc" Nov 01 20:20:33.909645 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:33.909615 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-933245-7cd656f7b9-7lszc"] Nov 01 20:20:33.910983 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:20:33.910953 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f516592_de0d_441d_aa8a_85a22568a94b.slice/crio-8d99b10dae328ef04e7a61f0b55162ce3387ace108b938b9367d8aca80b522e7 WatchSource:0}: Error finding container 8d99b10dae328ef04e7a61f0b55162ce3387ace108b938b9367d8aca80b522e7: Status 404 returned error can't find the container with id 8d99b10dae328ef04e7a61f0b55162ce3387ace108b938b9367d8aca80b522e7 Nov 01 20:20:33.912494 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:33.912473 5935 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 20:20:33.988212 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:33.988175 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-933245-7cd656f7b9-7lszc" event={"ID":"7f516592-de0d-441d-aa8a-85a22568a94b","Type":"ContainerStarted","Data":"8d99b10dae328ef04e7a61f0b55162ce3387ace108b938b9367d8aca80b522e7"} Nov 01 20:20:34.994511 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:34.994475 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-933245-7cd656f7b9-7lszc" event={"ID":"7f516592-de0d-441d-aa8a-85a22568a94b","Type":"ContainerStarted","Data":"fbd39275987feb181b5583a806732fab397cda040f38782352f5dd97a3d695ca"} Nov 01 20:20:34.994957 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:34.994556 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-933245-7cd656f7b9-7lszc" Nov 01 20:20:35.018368 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:35.018318 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-933245-7cd656f7b9-7lszc" podStartSLOduration=1.591533837 podStartE2EDuration="2.018303293s" totalImagesPullingTime="426.769456ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:20:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:20:35.011370639 +0000 UTC m=+1280.784531152" watchObservedRunningTime="2026-11-01 20:20:35.018303293 +0000 UTC m=+1280.791463799" Nov 01 20:20:35.843990 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:20:35.843955 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:20:36.797662 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:36.797628 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-a6873b-79b7f97879-9dhts" Nov 01 20:20:37.116370 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:37.116268 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8b6cd6-85cd59544b-dm2wk"] Nov 01 20:20:37.121439 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:37.121414 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8b6cd6-85cd59544b-dm2wk" Nov 01 20:20:37.126682 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:37.126642 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-8b6cd6-serving-cert\"" Nov 01 20:20:37.131837 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:37.131814 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8b6cd6-85cd59544b-dm2wk"] Nov 01 20:20:37.186624 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:37.186596 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh8z9\" (UniqueName: \"kubernetes.io/projected/56d6fb4f-c372-490f-9dec-5b36a2ab21e5-kube-api-access-qh8z9\") pod \"maas-api-e2e-shared-a-w4-8b6cd6-85cd59544b-dm2wk\" (UID: \"56d6fb4f-c372-490f-9dec-5b36a2ab21e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8b6cd6-85cd59544b-dm2wk" Nov 01 20:20:37.186770 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:37.186651 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56d6fb4f-c372-490f-9dec-5b36a2ab21e5-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-8b6cd6-85cd59544b-dm2wk\" (UID: \"56d6fb4f-c372-490f-9dec-5b36a2ab21e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8b6cd6-85cd59544b-dm2wk" Nov 01 20:20:37.186770 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:37.186759 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56d6fb4f-c372-490f-9dec-5b36a2ab21e5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-8b6cd6-85cd59544b-dm2wk\" (UID: \"56d6fb4f-c372-490f-9dec-5b36a2ab21e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8b6cd6-85cd59544b-dm2wk" Nov 01 20:20:37.186884 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:37.186859 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56d6fb4f-c372-490f-9dec-5b36a2ab21e5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-8b6cd6-85cd59544b-dm2wk\" (UID: \"56d6fb4f-c372-490f-9dec-5b36a2ab21e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8b6cd6-85cd59544b-dm2wk" Nov 01 20:20:37.288715 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:37.288683 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh8z9\" (UniqueName: \"kubernetes.io/projected/56d6fb4f-c372-490f-9dec-5b36a2ab21e5-kube-api-access-qh8z9\") pod \"maas-api-e2e-shared-a-w4-8b6cd6-85cd59544b-dm2wk\" (UID: \"56d6fb4f-c372-490f-9dec-5b36a2ab21e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8b6cd6-85cd59544b-dm2wk" Nov 01 20:20:37.288912 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:37.288846 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56d6fb4f-c372-490f-9dec-5b36a2ab21e5-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-8b6cd6-85cd59544b-dm2wk\" (UID: \"56d6fb4f-c372-490f-9dec-5b36a2ab21e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8b6cd6-85cd59544b-dm2wk" Nov 01 20:20:37.288992 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:37.288943 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56d6fb4f-c372-490f-9dec-5b36a2ab21e5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-8b6cd6-85cd59544b-dm2wk\" (UID: \"56d6fb4f-c372-490f-9dec-5b36a2ab21e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8b6cd6-85cd59544b-dm2wk" Nov 01 20:20:37.289050 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:37.289021 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56d6fb4f-c372-490f-9dec-5b36a2ab21e5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-8b6cd6-85cd59544b-dm2wk\" (UID: \"56d6fb4f-c372-490f-9dec-5b36a2ab21e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8b6cd6-85cd59544b-dm2wk" Nov 01 20:20:37.289857 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:37.289826 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56d6fb4f-c372-490f-9dec-5b36a2ab21e5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-8b6cd6-85cd59544b-dm2wk\" (UID: \"56d6fb4f-c372-490f-9dec-5b36a2ab21e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8b6cd6-85cd59544b-dm2wk" Nov 01 20:20:37.291680 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:37.291647 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56d6fb4f-c372-490f-9dec-5b36a2ab21e5-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-8b6cd6-85cd59544b-dm2wk\" (UID: \"56d6fb4f-c372-490f-9dec-5b36a2ab21e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8b6cd6-85cd59544b-dm2wk" Nov 01 20:20:37.291838 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:37.291823 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56d6fb4f-c372-490f-9dec-5b36a2ab21e5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-8b6cd6-85cd59544b-dm2wk\" (UID: \"56d6fb4f-c372-490f-9dec-5b36a2ab21e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8b6cd6-85cd59544b-dm2wk" Nov 01 20:20:37.296947 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:37.296922 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh8z9\" (UniqueName: \"kubernetes.io/projected/56d6fb4f-c372-490f-9dec-5b36a2ab21e5-kube-api-access-qh8z9\") pod \"maas-api-e2e-shared-a-w4-8b6cd6-85cd59544b-dm2wk\" (UID: \"56d6fb4f-c372-490f-9dec-5b36a2ab21e5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8b6cd6-85cd59544b-dm2wk" Nov 01 20:20:37.445397 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:37.445366 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8b6cd6-85cd59544b-dm2wk" Nov 01 20:20:37.762536 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:37.762441 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8b6cd6-85cd59544b-dm2wk"] Nov 01 20:20:38.021733 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:38.021653 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8b6cd6-85cd59544b-dm2wk" event={"ID":"56d6fb4f-c372-490f-9dec-5b36a2ab21e5","Type":"ContainerStarted","Data":"95da55ab40226bec5bbae16d041b6875c5819222757c11315605ab4531b0b5a9"} Nov 01 20:20:39.029186 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:39.029142 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8b6cd6-85cd59544b-dm2wk" event={"ID":"56d6fb4f-c372-490f-9dec-5b36a2ab21e5","Type":"ContainerStarted","Data":"f5044c1bb2893658231a8f84cb20a7291bfcce2ac2aa001a923632caee0839a9"} Nov 01 20:20:39.029614 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:39.029297 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8b6cd6-85cd59544b-dm2wk" Nov 01 20:20:39.047110 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:39.047065 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8b6cd6-85cd59544b-dm2wk" podStartSLOduration=1.579976497 podStartE2EDuration="2.047052447s" totalImagesPullingTime="467.07595ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:20:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:20:39.044392502 +0000 UTC m=+1284.817553016" watchObservedRunningTime="2026-11-01 20:20:39.047052447 +0000 UTC m=+1284.820212999" Nov 01 20:20:41.010261 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:41.010231 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-933245-7cd656f7b9-7lszc" Nov 01 20:20:45.042805 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:45.042773 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8b6cd6-85cd59544b-dm2wk" Nov 01 20:20:59.266275 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:59.266241 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-036c69-79cd8c8b6f-fxpb5"] Nov 01 20:20:59.272031 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:59.272009 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-036c69-79cd8c8b6f-fxpb5" Nov 01 20:20:59.279079 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:59.279053 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-036c69-dockercfg-vb5c2\"" Nov 01 20:20:59.296923 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:59.290273 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-036c69-79cd8c8b6f-fxpb5"] Nov 01 20:20:59.339011 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:59.338981 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/624bb371-6040-4cd4-a668-a8afc016d132-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-036c69-79cd8c8b6f-fxpb5\" (UID: \"624bb371-6040-4cd4-a668-a8afc016d132\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-036c69-79cd8c8b6f-fxpb5" Nov 01 20:20:59.339144 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:59.339028 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5hnt\" (UniqueName: \"kubernetes.io/projected/624bb371-6040-4cd4-a668-a8afc016d132-kube-api-access-f5hnt\") pod \"payload-processing-e2e-shared-b-w4-036c69-79cd8c8b6f-fxpb5\" (UID: \"624bb371-6040-4cd4-a668-a8afc016d132\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-036c69-79cd8c8b6f-fxpb5" Nov 01 20:20:59.351930 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:59.351904 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-036c69\"" Nov 01 20:20:59.446172 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:59.446128 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/624bb371-6040-4cd4-a668-a8afc016d132-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-036c69-79cd8c8b6f-fxpb5\" (UID: \"624bb371-6040-4cd4-a668-a8afc016d132\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-036c69-79cd8c8b6f-fxpb5" Nov 01 20:20:59.446392 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:59.446202 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5hnt\" (UniqueName: \"kubernetes.io/projected/624bb371-6040-4cd4-a668-a8afc016d132-kube-api-access-f5hnt\") pod \"payload-processing-e2e-shared-b-w4-036c69-79cd8c8b6f-fxpb5\" (UID: \"624bb371-6040-4cd4-a668-a8afc016d132\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-036c69-79cd8c8b6f-fxpb5" Nov 01 20:20:59.448197 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:59.447931 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/624bb371-6040-4cd4-a668-a8afc016d132-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-036c69-79cd8c8b6f-fxpb5\" (UID: \"624bb371-6040-4cd4-a668-a8afc016d132\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-036c69-79cd8c8b6f-fxpb5" Nov 01 20:20:59.457121 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:59.457064 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5hnt\" (UniqueName: \"kubernetes.io/projected/624bb371-6040-4cd4-a668-a8afc016d132-kube-api-access-f5hnt\") pod \"payload-processing-e2e-shared-b-w4-036c69-79cd8c8b6f-fxpb5\" (UID: \"624bb371-6040-4cd4-a668-a8afc016d132\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-036c69-79cd8c8b6f-fxpb5" Nov 01 20:20:59.591843 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:59.591755 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-036c69-79cd8c8b6f-fxpb5" Nov 01 20:20:59.881678 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:20:59.881648 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-036c69-79cd8c8b6f-fxpb5"] Nov 01 20:20:59.883007 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:20:59.882978 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod624bb371_6040_4cd4_a668_a8afc016d132.slice/crio-18c917bb9be838675304ecb6a613727e657cdcc08c3b0068819c43c2428174ad WatchSource:0}: Error finding container 18c917bb9be838675304ecb6a613727e657cdcc08c3b0068819c43c2428174ad: Status 404 returned error can't find the container with id 18c917bb9be838675304ecb6a613727e657cdcc08c3b0068819c43c2428174ad Nov 01 20:21:00.210346 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:00.209027 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-036c69-79cd8c8b6f-fxpb5" event={"ID":"624bb371-6040-4cd4-a668-a8afc016d132","Type":"ContainerStarted","Data":"fe2827cfb5d9c4adf9c8cb52d85dcfafbd21e70d88b11f1fa46cd667fe43c447"} Nov 01 20:21:00.210346 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:00.209096 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-036c69-79cd8c8b6f-fxpb5" event={"ID":"624bb371-6040-4cd4-a668-a8afc016d132","Type":"ContainerStarted","Data":"18c917bb9be838675304ecb6a613727e657cdcc08c3b0068819c43c2428174ad"} Nov 01 20:21:00.210346 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:00.210270 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-036c69-79cd8c8b6f-fxpb5" Nov 01 20:21:00.245339 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:00.238366 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-036c69-79cd8c8b6f-fxpb5" podStartSLOduration=1.238346947 podStartE2EDuration="1.238346947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:20:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:21:00.235680753 +0000 UTC m=+1306.008841267" watchObservedRunningTime="2026-11-01 20:21:00.238346947 +0000 UTC m=+1306.011507460" Nov 01 20:21:05.846683 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:21:05.846637 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:21:11.504752 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:11.504714 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-bd79chz4c"] Nov 01 20:21:11.510101 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:11.510078 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-bd79chz4c" Nov 01 20:21:11.516843 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:11.516814 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-bd79chz4c"] Nov 01 20:21:11.518890 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:11.518870 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-aaea41\"/\"routing-overlay\"" Nov 01 20:21:11.519958 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:11.519940 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-aaea41\"/\"agc-cred-8876b9054304\"" Nov 01 20:21:11.520043 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:11.520006 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-aaea41\"/\"agc-cred-85078fdb41c3\"" Nov 01 20:21:11.520043 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:11.520015 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-aaea41\"/\"kube-root-ca.crt\"" Nov 01 20:21:11.520151 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:11.520087 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-aaea41\"/\"openshift-service-ca.crt\"" Nov 01 20:21:11.520203 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:11.520178 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-aaea41\"/\"payload-processing-external-model-e2e--245da986-dockercfg-r9b8m\"" Nov 01 20:21:11.520256 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:11.520240 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-aaea41\"/\"e2e-cfl-e2318b-provider-cred\"" Nov 01 20:21:11.520330 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:11.520317 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-aaea41\"/\"payload-processing-external-model-plugins-e2e-extmod-a-aaea41\"" Nov 01 20:21:11.607970 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:11.607929 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/eb42ea95-05d4-4571-b24e-00079eee0a22-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-bd79chz4c\" (UID: \"eb42ea95-05d4-4571-b24e-00079eee0a22\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-bd79chz4c" Nov 01 20:21:11.608157 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:11.608003 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dpzx\" (UniqueName: \"kubernetes.io/projected/eb42ea95-05d4-4571-b24e-00079eee0a22-kube-api-access-8dpzx\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-bd79chz4c\" (UID: \"eb42ea95-05d4-4571-b24e-00079eee0a22\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-bd79chz4c" Nov 01 20:21:11.609394 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:11.609361 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/eb42ea95-05d4-4571-b24e-00079eee0a22-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-bd79chz4c\" (UID: \"eb42ea95-05d4-4571-b24e-00079eee0a22\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-bd79chz4c" Nov 01 20:21:11.610119 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:11.609562 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb42ea95-05d4-4571-b24e-00079eee0a22-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-bd79chz4c\" (UID: \"eb42ea95-05d4-4571-b24e-00079eee0a22\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-bd79chz4c" Nov 01 20:21:11.710886 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:11.710857 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb42ea95-05d4-4571-b24e-00079eee0a22-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-bd79chz4c\" (UID: \"eb42ea95-05d4-4571-b24e-00079eee0a22\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-bd79chz4c" Nov 01 20:21:11.711051 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:11.710980 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/eb42ea95-05d4-4571-b24e-00079eee0a22-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-bd79chz4c\" (UID: \"eb42ea95-05d4-4571-b24e-00079eee0a22\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-bd79chz4c" Nov 01 20:21:11.711051 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:11.711014 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dpzx\" (UniqueName: \"kubernetes.io/projected/eb42ea95-05d4-4571-b24e-00079eee0a22-kube-api-access-8dpzx\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-bd79chz4c\" (UID: \"eb42ea95-05d4-4571-b24e-00079eee0a22\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-bd79chz4c" Nov 01 20:21:11.711140 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:11.711080 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/eb42ea95-05d4-4571-b24e-00079eee0a22-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-bd79chz4c\" (UID: \"eb42ea95-05d4-4571-b24e-00079eee0a22\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-bd79chz4c" Nov 01 20:21:11.711684 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:11.711660 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/eb42ea95-05d4-4571-b24e-00079eee0a22-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-bd79chz4c\" (UID: \"eb42ea95-05d4-4571-b24e-00079eee0a22\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-bd79chz4c" Nov 01 20:21:11.711684 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:11.711677 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb42ea95-05d4-4571-b24e-00079eee0a22-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-bd79chz4c\" (UID: \"eb42ea95-05d4-4571-b24e-00079eee0a22\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-bd79chz4c" Nov 01 20:21:11.713777 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:11.713750 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/eb42ea95-05d4-4571-b24e-00079eee0a22-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-bd79chz4c\" (UID: \"eb42ea95-05d4-4571-b24e-00079eee0a22\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-bd79chz4c" Nov 01 20:21:11.718679 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:11.718656 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dpzx\" (UniqueName: \"kubernetes.io/projected/eb42ea95-05d4-4571-b24e-00079eee0a22-kube-api-access-8dpzx\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-bd79chz4c\" (UID: \"eb42ea95-05d4-4571-b24e-00079eee0a22\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-bd79chz4c" Nov 01 20:21:11.828857 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:11.828764 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-bd79chz4c" Nov 01 20:21:12.132202 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:12.132174 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-bd79chz4c"] Nov 01 20:21:12.133775 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:21:12.133750 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb42ea95_05d4_4571_b24e_00079eee0a22.slice/crio-3edd5f972c378e9ce49db791b25ddbc5985a29acdeacbba6627b1dc31c6e8f3f WatchSource:0}: Error finding container 3edd5f972c378e9ce49db791b25ddbc5985a29acdeacbba6627b1dc31c6e8f3f: Status 404 returned error can't find the container with id 3edd5f972c378e9ce49db791b25ddbc5985a29acdeacbba6627b1dc31c6e8f3f Nov 01 20:21:12.227059 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:12.227033 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-036c69-79cd8c8b6f-fxpb5" Nov 01 20:21:12.300698 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:12.300661 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-bd79chz4c" event={"ID":"eb42ea95-05d4-4571-b24e-00079eee0a22","Type":"ContainerStarted","Data":"3ff3c0bcaad2b0748f4282e3a96a1555b27576f079bd57d955aae4e13629ed6a"} Nov 01 20:21:12.300897 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:12.300736 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-bd79chz4c" Nov 01 20:21:12.300897 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:12.300763 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-bd79chz4c" event={"ID":"eb42ea95-05d4-4571-b24e-00079eee0a22","Type":"ContainerStarted","Data":"3edd5f972c378e9ce49db791b25ddbc5985a29acdeacbba6627b1dc31c6e8f3f"} Nov 01 20:21:12.321380 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:12.321312 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-bd79chz4c" podStartSLOduration=1.321294413 podStartE2EDuration="1.321294413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:21:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:21:12.321042266 +0000 UTC m=+1318.094202803" watchObservedRunningTime="2026-11-01 20:21:12.321294413 +0000 UTC m=+1318.094454918" Nov 01 20:21:12.511740 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:12.511645 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-bd79chz4c"] Nov 01 20:21:13.339417 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:21:13.339372 5935 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-aaea41/agc-cred-85078fdb41c3: secret "agc-cred-85078fdb41c3" not found Nov 01 20:21:13.339417 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:21:13.339408 5935 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-bd79chz4c: secret "agc-cred-85078fdb41c3" not found Nov 01 20:21:13.339685 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:21:13.339467 5935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb42ea95-05d4-4571-b24e-00079eee0a22-provider-credentials podName:eb42ea95-05d4-4571-b24e-00079eee0a22 nodeName:}" failed. No retries permitted until 2026-11-01 20:21:13.839448318 +0000 UTC m=+1319.612608807 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/eb42ea95-05d4-4571-b24e-00079eee0a22-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-aaea41-bd79chz4c" (UID: "eb42ea95-05d4-4571-b24e-00079eee0a22") : secret "agc-cred-85078fdb41c3" not found Nov 01 20:21:13.846348 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:21:13.846312 5935 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-aaea41/agc-cred-85078fdb41c3: secret "agc-cred-85078fdb41c3" not found Nov 01 20:21:13.846348 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:21:13.846349 5935 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-bd79chz4c: secret "agc-cred-85078fdb41c3" not found Nov 01 20:21:13.846859 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:21:13.846413 5935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb42ea95-05d4-4571-b24e-00079eee0a22-provider-credentials podName:eb42ea95-05d4-4571-b24e-00079eee0a22 nodeName:}" failed. No retries permitted until 2026-11-01 20:21:14.846395478 +0000 UTC m=+1320.619555987 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/eb42ea95-05d4-4571-b24e-00079eee0a22-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-aaea41-bd79chz4c" (UID: "eb42ea95-05d4-4571-b24e-00079eee0a22") : secret "agc-cred-85078fdb41c3" not found Nov 01 20:21:14.870577 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:21:14.870545 5935 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-aaea41/agc-cred-85078fdb41c3: secret "agc-cred-85078fdb41c3" not found Nov 01 20:21:14.870577 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:21:14.870579 5935 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-bd79chz4c: secret "agc-cred-85078fdb41c3" not found Nov 01 20:21:14.870973 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:21:14.870632 5935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb42ea95-05d4-4571-b24e-00079eee0a22-provider-credentials podName:eb42ea95-05d4-4571-b24e-00079eee0a22 nodeName:}" failed. No retries permitted until 2026-11-01 20:21:16.870618607 +0000 UTC m=+1322.643779112 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/eb42ea95-05d4-4571-b24e-00079eee0a22-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-aaea41-bd79chz4c" (UID: "eb42ea95-05d4-4571-b24e-00079eee0a22") : secret "agc-cred-85078fdb41c3" not found Nov 01 20:21:16.907775 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:21:16.907740 5935 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-aaea41/agc-cred-85078fdb41c3: secret "agc-cred-85078fdb41c3" not found Nov 01 20:21:16.908374 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:21:16.908346 5935 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-bd79chz4c: secret "agc-cred-85078fdb41c3" not found Nov 01 20:21:16.908500 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:21:16.908437 5935 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/eb42ea95-05d4-4571-b24e-00079eee0a22-provider-credentials podName:eb42ea95-05d4-4571-b24e-00079eee0a22 nodeName:}" failed. No retries permitted until 2026-11-01 20:21:20.908413395 +0000 UTC m=+1326.681573885 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/eb42ea95-05d4-4571-b24e-00079eee0a22-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-aaea41-bd79chz4c" (UID: "eb42ea95-05d4-4571-b24e-00079eee0a22") : secret "agc-cred-85078fdb41c3" not found Nov 01 20:21:19.314714 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:19.314661 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-bd79chz4c" podUID="eb42ea95-05d4-4571-b24e-00079eee0a22" containerName="payload-processing" containerID="cri-o://3ff3c0bcaad2b0748f4282e3a96a1555b27576f079bd57d955aae4e13629ed6a" gracePeriod=25 Nov 01 20:21:19.602133 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:19.602111 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-bd79chz4c" Nov 01 20:21:19.753015 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:19.752973 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb42ea95-05d4-4571-b24e-00079eee0a22-provider-credentials\" (UniqueName: \"kubernetes.io/projected/eb42ea95-05d4-4571-b24e-00079eee0a22-provider-credentials\") pod \"eb42ea95-05d4-4571-b24e-00079eee0a22\" (UID: \"eb42ea95-05d4-4571-b24e-00079eee0a22\") " Nov 01 20:21:19.753211 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:19.753032 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb42ea95-05d4-4571-b24e-00079eee0a22-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb42ea95-05d4-4571-b24e-00079eee0a22-plugins-config-volume\") pod \"eb42ea95-05d4-4571-b24e-00079eee0a22\" (UID: \"eb42ea95-05d4-4571-b24e-00079eee0a22\") " Nov 01 20:21:19.753211 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:19.753108 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb42ea95-05d4-4571-b24e-00079eee0a22-kube-api-access-8dpzx\" (UniqueName: \"kubernetes.io/projected/eb42ea95-05d4-4571-b24e-00079eee0a22-kube-api-access-8dpzx\") pod \"eb42ea95-05d4-4571-b24e-00079eee0a22\" (UID: \"eb42ea95-05d4-4571-b24e-00079eee0a22\") " Nov 01 20:21:19.753385 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:19.753230 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb42ea95-05d4-4571-b24e-00079eee0a22-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/eb42ea95-05d4-4571-b24e-00079eee0a22-routing-overlay-volume\") pod \"eb42ea95-05d4-4571-b24e-00079eee0a22\" (UID: \"eb42ea95-05d4-4571-b24e-00079eee0a22\") " Nov 01 20:21:19.753490 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:19.753466 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb42ea95-05d4-4571-b24e-00079eee0a22-plugins-config-volume" pod "eb42ea95-05d4-4571-b24e-00079eee0a22" (UID: "eb42ea95-05d4-4571-b24e-00079eee0a22"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:21:19.753824 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:19.753791 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb42ea95-05d4-4571-b24e-00079eee0a22-routing-overlay-volume" pod "eb42ea95-05d4-4571-b24e-00079eee0a22" (UID: "eb42ea95-05d4-4571-b24e-00079eee0a22"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:21:19.754036 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:19.754014 5935 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb42ea95-05d4-4571-b24e-00079eee0a22-plugins-config-volume\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:21:19.755610 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:19.755585 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb42ea95-05d4-4571-b24e-00079eee0a22-provider-credentials" pod "eb42ea95-05d4-4571-b24e-00079eee0a22" (UID: "eb42ea95-05d4-4571-b24e-00079eee0a22"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:21:19.755686 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:19.755624 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb42ea95-05d4-4571-b24e-00079eee0a22-kube-api-access-8dpzx" pod "eb42ea95-05d4-4571-b24e-00079eee0a22" (UID: "eb42ea95-05d4-4571-b24e-00079eee0a22"). InnerVolumeSpecName "kube-api-access-8dpzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:21:19.854962 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:19.854892 5935 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/eb42ea95-05d4-4571-b24e-00079eee0a22-routing-overlay-volume\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:21:19.854962 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:19.854919 5935 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/eb42ea95-05d4-4571-b24e-00079eee0a22-provider-credentials\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:21:19.854962 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:19.854935 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dpzx\" (UniqueName: \"kubernetes.io/projected/eb42ea95-05d4-4571-b24e-00079eee0a22-kube-api-access-8dpzx\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:21:20.375093 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:20.375063 5935 generic.go:356] "Generic (PLEG): container finished" podID="eb42ea95-05d4-4571-b24e-00079eee0a22" containerID="3ff3c0bcaad2b0748f4282e3a96a1555b27576f079bd57d955aae4e13629ed6a" exitCode=0 Nov 01 20:21:20.375520 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:20.375097 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-bd79chz4c" Nov 01 20:21:20.375520 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:20.375146 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-bd79chz4c" event={"ID":"eb42ea95-05d4-4571-b24e-00079eee0a22","Type":"ContainerDied","Data":"3ff3c0bcaad2b0748f4282e3a96a1555b27576f079bd57d955aae4e13629ed6a"} Nov 01 20:21:20.375520 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:20.375199 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-bd79chz4c" event={"ID":"eb42ea95-05d4-4571-b24e-00079eee0a22","Type":"ContainerDied","Data":"3edd5f972c378e9ce49db791b25ddbc5985a29acdeacbba6627b1dc31c6e8f3f"} Nov 01 20:21:20.375520 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:20.375222 5935 scope.go:122] "RemoveContainer" containerID="3ff3c0bcaad2b0748f4282e3a96a1555b27576f079bd57d955aae4e13629ed6a" Nov 01 20:21:20.385173 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:20.385156 5935 scope.go:122] "RemoveContainer" containerID="3ff3c0bcaad2b0748f4282e3a96a1555b27576f079bd57d955aae4e13629ed6a" Nov 01 20:21:20.385456 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:21:20.385437 5935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ff3c0bcaad2b0748f4282e3a96a1555b27576f079bd57d955aae4e13629ed6a\": container with ID starting with 3ff3c0bcaad2b0748f4282e3a96a1555b27576f079bd57d955aae4e13629ed6a not found: ID does not exist" containerID="3ff3c0bcaad2b0748f4282e3a96a1555b27576f079bd57d955aae4e13629ed6a" Nov 01 20:21:20.385528 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:20.385463 5935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ff3c0bcaad2b0748f4282e3a96a1555b27576f079bd57d955aae4e13629ed6a"} err="failed to get container status \"3ff3c0bcaad2b0748f4282e3a96a1555b27576f079bd57d955aae4e13629ed6a\": rpc error: code = NotFound desc = could not find container \"3ff3c0bcaad2b0748f4282e3a96a1555b27576f079bd57d955aae4e13629ed6a\": container with ID starting with 3ff3c0bcaad2b0748f4282e3a96a1555b27576f079bd57d955aae4e13629ed6a not found: ID does not exist" Nov 01 20:21:20.393178 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:20.393121 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-bd79chz4c"] Nov 01 20:21:20.397112 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:20.397085 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-bd79chz4c"] Nov 01 20:21:22.354097 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:22.354066 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb42ea95-05d4-4571-b24e-00079eee0a22" path="/var/lib/kubelet/pods/eb42ea95-05d4-4571-b24e-00079eee0a22/volumes" Nov 01 20:21:25.352438 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:25.352404 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-6976p66fm"] Nov 01 20:21:25.353206 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:25.353185 5935 cpu_manager.go:414] "containerMap: removing container" podUID="eb42ea95-05d4-4571-b24e-00079eee0a22" containerName="payload-processing" Nov 01 20:21:25.353336 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:25.353207 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb42ea95-05d4-4571-b24e-00079eee0a22" containerName="payload-processing" Nov 01 20:21:25.353436 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:25.353422 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb42ea95-05d4-4571-b24e-00079eee0a22" containerName="payload-processing" Nov 01 20:21:25.358355 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:25.358333 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-6976p66fm" Nov 01 20:21:25.366266 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:25.366239 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-6976p66fm"] Nov 01 20:21:25.378591 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:25.378564 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-aaea41\"/\"openshift-service-ca.crt\"" Nov 01 20:21:25.379980 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:25.379957 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-aaea41\"/\"e2e-cfl-e2318b-provider-cred\"" Nov 01 20:21:25.380223 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:25.380205 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-aaea41\"/\"payload-processing-external-model-plugins-e2e-extmod-a-aaea41\"" Nov 01 20:21:25.380538 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:25.380460 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-aaea41\"/\"payload-processing-external-model-e2e--245da986-dockercfg-r9b8m\"" Nov 01 20:21:25.380837 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:25.380820 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-aaea41\"/\"routing-overlay\"" Nov 01 20:21:25.381014 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:25.380999 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-aaea41\"/\"kube-root-ca.crt\"" Nov 01 20:21:25.491208 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:25.490547 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8f8d270f-189b-4434-b911-368d1082cb97-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-6976p66fm\" (UID: \"8f8d270f-189b-4434-b911-368d1082cb97\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-6976p66fm" Nov 01 20:21:25.491208 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:25.490620 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcv9c\" (UniqueName: \"kubernetes.io/projected/8f8d270f-189b-4434-b911-368d1082cb97-kube-api-access-qcv9c\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-6976p66fm\" (UID: \"8f8d270f-189b-4434-b911-368d1082cb97\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-6976p66fm" Nov 01 20:21:25.491208 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:25.490778 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f8d270f-189b-4434-b911-368d1082cb97-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-6976p66fm\" (UID: \"8f8d270f-189b-4434-b911-368d1082cb97\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-6976p66fm" Nov 01 20:21:25.491208 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:25.490807 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8f8d270f-189b-4434-b911-368d1082cb97-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-6976p66fm\" (UID: \"8f8d270f-189b-4434-b911-368d1082cb97\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-6976p66fm" Nov 01 20:21:25.591794 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:25.591762 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8f8d270f-189b-4434-b911-368d1082cb97-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-6976p66fm\" (UID: \"8f8d270f-189b-4434-b911-368d1082cb97\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-6976p66fm" Nov 01 20:21:25.591967 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:25.591810 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcv9c\" (UniqueName: \"kubernetes.io/projected/8f8d270f-189b-4434-b911-368d1082cb97-kube-api-access-qcv9c\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-6976p66fm\" (UID: \"8f8d270f-189b-4434-b911-368d1082cb97\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-6976p66fm" Nov 01 20:21:25.591967 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:25.591893 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f8d270f-189b-4434-b911-368d1082cb97-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-6976p66fm\" (UID: \"8f8d270f-189b-4434-b911-368d1082cb97\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-6976p66fm" Nov 01 20:21:25.592061 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:25.592000 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8f8d270f-189b-4434-b911-368d1082cb97-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-6976p66fm\" (UID: \"8f8d270f-189b-4434-b911-368d1082cb97\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-6976p66fm" Nov 01 20:21:25.592530 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:25.592509 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8f8d270f-189b-4434-b911-368d1082cb97-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-6976p66fm\" (UID: \"8f8d270f-189b-4434-b911-368d1082cb97\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-6976p66fm" Nov 01 20:21:25.592652 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:25.592635 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f8d270f-189b-4434-b911-368d1082cb97-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-6976p66fm\" (UID: \"8f8d270f-189b-4434-b911-368d1082cb97\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-6976p66fm" Nov 01 20:21:25.594504 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:25.594481 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8f8d270f-189b-4434-b911-368d1082cb97-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-6976p66fm\" (UID: \"8f8d270f-189b-4434-b911-368d1082cb97\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-6976p66fm" Nov 01 20:21:25.599838 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:25.599816 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcv9c\" (UniqueName: \"kubernetes.io/projected/8f8d270f-189b-4434-b911-368d1082cb97-kube-api-access-qcv9c\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-6976p66fm\" (UID: \"8f8d270f-189b-4434-b911-368d1082cb97\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-6976p66fm" Nov 01 20:21:25.734141 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:25.734114 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-6976p66fm" Nov 01 20:21:26.011009 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:26.010976 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-6976p66fm"] Nov 01 20:21:26.012269 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:21:26.012242 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f8d270f_189b_4434_b911_368d1082cb97.slice/crio-156ab16f0b868918ee4b7a71bd9ce968dcf0dfe0d74e7c27acaf01de1092a04a WatchSource:0}: Error finding container 156ab16f0b868918ee4b7a71bd9ce968dcf0dfe0d74e7c27acaf01de1092a04a: Status 404 returned error can't find the container with id 156ab16f0b868918ee4b7a71bd9ce968dcf0dfe0d74e7c27acaf01de1092a04a Nov 01 20:21:26.484066 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:26.484033 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-6976p66fm" event={"ID":"8f8d270f-189b-4434-b911-368d1082cb97","Type":"ContainerStarted","Data":"6dbc64221f2677dd580f5f674ff844aaee0c740cd005a6aaedc8ad40d825fd1f"} Nov 01 20:21:26.484577 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:26.484090 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-6976p66fm" event={"ID":"8f8d270f-189b-4434-b911-368d1082cb97","Type":"ContainerStarted","Data":"156ab16f0b868918ee4b7a71bd9ce968dcf0dfe0d74e7c27acaf01de1092a04a"} Nov 01 20:21:26.484577 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:26.484234 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-6976p66fm" Nov 01 20:21:26.499942 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:26.499902 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-6976p66fm" podStartSLOduration=1.4998915099999999 podStartE2EDuration="1.49989151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:21:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:21:26.498634172 +0000 UTC m=+1332.271794685" watchObservedRunningTime="2026-11-01 20:21:26.49989151 +0000 UTC m=+1332.273052056" Nov 01 20:21:35.844971 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:21:35.844927 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:21:36.867008 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:36.866973 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-6976p66fm"] Nov 01 20:21:36.872030 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:36.871997 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-6976p66fm" Nov 01 20:21:41.868208 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:41.868167 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-6976p66fm" podUID="8f8d270f-189b-4434-b911-368d1082cb97" containerName="payload-processing" containerID="cri-o://6dbc64221f2677dd580f5f674ff844aaee0c740cd005a6aaedc8ad40d825fd1f" gracePeriod=25 Nov 01 20:21:42.139003 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:42.138975 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-6976p66fm" Nov 01 20:21:42.213128 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:42.213084 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w"] Nov 01 20:21:42.213348 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:42.213320 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w" podUID="3118450b-b033-4def-9ccd-39547f6602f4" containerName="istio-proxy" containerID="cri-o://2a8ea06c755e317176ba32e3fd0bb4c75fd4414b205f92f1642cad3f0d618348" gracePeriod=30 Nov 01 20:21:42.265865 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:42.265838 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f8d270f-189b-4434-b911-368d1082cb97-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8f8d270f-189b-4434-b911-368d1082cb97-routing-overlay-volume\") pod \"8f8d270f-189b-4434-b911-368d1082cb97\" (UID: \"8f8d270f-189b-4434-b911-368d1082cb97\") " Nov 01 20:21:42.266030 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:42.265885 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f8d270f-189b-4434-b911-368d1082cb97-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8f8d270f-189b-4434-b911-368d1082cb97-provider-credentials\") pod \"8f8d270f-189b-4434-b911-368d1082cb97\" (UID: \"8f8d270f-189b-4434-b911-368d1082cb97\") " Nov 01 20:21:42.266129 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:42.266111 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f8d270f-189b-4434-b911-368d1082cb97-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f8d270f-189b-4434-b911-368d1082cb97-plugins-config-volume\") pod \"8f8d270f-189b-4434-b911-368d1082cb97\" (UID: \"8f8d270f-189b-4434-b911-368d1082cb97\") " Nov 01 20:21:42.266225 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:42.266204 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f8d270f-189b-4434-b911-368d1082cb97-kube-api-access-qcv9c\" (UniqueName: \"kubernetes.io/projected/8f8d270f-189b-4434-b911-368d1082cb97-kube-api-access-qcv9c\") pod \"8f8d270f-189b-4434-b911-368d1082cb97\" (UID: \"8f8d270f-189b-4434-b911-368d1082cb97\") " Nov 01 20:21:42.266322 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:42.266233 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f8d270f-189b-4434-b911-368d1082cb97-routing-overlay-volume" pod "8f8d270f-189b-4434-b911-368d1082cb97" (UID: "8f8d270f-189b-4434-b911-368d1082cb97"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:21:42.266587 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:42.266560 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f8d270f-189b-4434-b911-368d1082cb97-plugins-config-volume" pod "8f8d270f-189b-4434-b911-368d1082cb97" (UID: "8f8d270f-189b-4434-b911-368d1082cb97"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:21:42.266817 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:42.266797 5935 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8f8d270f-189b-4434-b911-368d1082cb97-routing-overlay-volume\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:21:42.266871 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:42.266822 5935 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f8d270f-189b-4434-b911-368d1082cb97-plugins-config-volume\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:21:42.268022 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:42.267999 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f8d270f-189b-4434-b911-368d1082cb97-provider-credentials" pod "8f8d270f-189b-4434-b911-368d1082cb97" (UID: "8f8d270f-189b-4434-b911-368d1082cb97"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:21:42.268156 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:42.268136 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f8d270f-189b-4434-b911-368d1082cb97-kube-api-access-qcv9c" pod "8f8d270f-189b-4434-b911-368d1082cb97" (UID: "8f8d270f-189b-4434-b911-368d1082cb97"). InnerVolumeSpecName "kube-api-access-qcv9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:21:42.367636 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:42.367609 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcv9c\" (UniqueName: \"kubernetes.io/projected/8f8d270f-189b-4434-b911-368d1082cb97-kube-api-access-qcv9c\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:21:42.367636 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:42.367635 5935 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8f8d270f-189b-4434-b911-368d1082cb97-provider-credentials\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:21:42.424507 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:42.424443 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56"] Nov 01 20:21:42.424653 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:42.424629 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56" podUID="1fe8488d-838d-47fe-b2c5-8b88dfb5724e" containerName="istio-proxy" containerID="cri-o://122796766cd7ef50fe4d4c7a2ace79ee85d0bb38c329595b5333b9553653dd49" gracePeriod=30 Nov 01 20:21:42.637943 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:42.637901 5935 generic.go:356] "Generic (PLEG): container finished" podID="8f8d270f-189b-4434-b911-368d1082cb97" containerID="6dbc64221f2677dd580f5f674ff844aaee0c740cd005a6aaedc8ad40d825fd1f" exitCode=0 Nov 01 20:21:42.638143 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:42.638003 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-6976p66fm" Nov 01 20:21:42.638143 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:42.638003 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-6976p66fm" event={"ID":"8f8d270f-189b-4434-b911-368d1082cb97","Type":"ContainerDied","Data":"6dbc64221f2677dd580f5f674ff844aaee0c740cd005a6aaedc8ad40d825fd1f"} Nov 01 20:21:42.638821 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:42.638202 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-6976p66fm" event={"ID":"8f8d270f-189b-4434-b911-368d1082cb97","Type":"ContainerDied","Data":"156ab16f0b868918ee4b7a71bd9ce968dcf0dfe0d74e7c27acaf01de1092a04a"} Nov 01 20:21:42.638821 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:42.638244 5935 scope.go:122] "RemoveContainer" containerID="6dbc64221f2677dd580f5f674ff844aaee0c740cd005a6aaedc8ad40d825fd1f" Nov 01 20:21:42.652213 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:42.652191 5935 scope.go:122] "RemoveContainer" containerID="6dbc64221f2677dd580f5f674ff844aaee0c740cd005a6aaedc8ad40d825fd1f" Nov 01 20:21:42.652509 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:21:42.652487 5935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dbc64221f2677dd580f5f674ff844aaee0c740cd005a6aaedc8ad40d825fd1f\": container with ID starting with 6dbc64221f2677dd580f5f674ff844aaee0c740cd005a6aaedc8ad40d825fd1f not found: ID does not exist" containerID="6dbc64221f2677dd580f5f674ff844aaee0c740cd005a6aaedc8ad40d825fd1f" Nov 01 20:21:42.652597 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:42.652521 5935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dbc64221f2677dd580f5f674ff844aaee0c740cd005a6aaedc8ad40d825fd1f"} err="failed to get container status \"6dbc64221f2677dd580f5f674ff844aaee0c740cd005a6aaedc8ad40d825fd1f\": rpc error: code = NotFound desc = could not find container \"6dbc64221f2677dd580f5f674ff844aaee0c740cd005a6aaedc8ad40d825fd1f\": container with ID starting with 6dbc64221f2677dd580f5f674ff844aaee0c740cd005a6aaedc8ad40d825fd1f not found: ID does not exist" Nov 01 20:21:42.658027 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:42.658001 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-6976p66fm"] Nov 01 20:21:42.661058 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:42.661034 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-6976p66fm"] Nov 01 20:21:44.354629 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:44.354594 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f8d270f-189b-4434-b911-368d1082cb97" path="/var/lib/kubelet/pods/8f8d270f-189b-4434-b911-368d1082cb97/volumes" Nov 01 20:21:47.537759 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:47.537728 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w" Nov 01 20:21:47.644486 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:47.644454 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3118450b-b033-4def-9ccd-39547f6602f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3118450b-b033-4def-9ccd-39547f6602f4-istio-envoy\") pod \"3118450b-b033-4def-9ccd-39547f6602f4\" (UID: \"3118450b-b033-4def-9ccd-39547f6602f4\") " Nov 01 20:21:47.644486 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:47.644491 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3118450b-b033-4def-9ccd-39547f6602f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3118450b-b033-4def-9ccd-39547f6602f4-workload-socket\") pod \"3118450b-b033-4def-9ccd-39547f6602f4\" (UID: \"3118450b-b033-4def-9ccd-39547f6602f4\") " Nov 01 20:21:47.644683 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:47.644510 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3118450b-b033-4def-9ccd-39547f6602f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3118450b-b033-4def-9ccd-39547f6602f4-credential-socket\") pod \"3118450b-b033-4def-9ccd-39547f6602f4\" (UID: \"3118450b-b033-4def-9ccd-39547f6602f4\") " Nov 01 20:21:47.644683 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:47.644620 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3118450b-b033-4def-9ccd-39547f6602f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3118450b-b033-4def-9ccd-39547f6602f4-istio-data\") pod \"3118450b-b033-4def-9ccd-39547f6602f4\" (UID: \"3118450b-b033-4def-9ccd-39547f6602f4\") " Nov 01 20:21:47.644769 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:47.644737 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3118450b-b033-4def-9ccd-39547f6602f4-workload-socket" pod "3118450b-b033-4def-9ccd-39547f6602f4" (UID: "3118450b-b033-4def-9ccd-39547f6602f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:21:47.644769 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:47.644757 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3118450b-b033-4def-9ccd-39547f6602f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3118450b-b033-4def-9ccd-39547f6602f4-workload-certs\") pod \"3118450b-b033-4def-9ccd-39547f6602f4\" (UID: \"3118450b-b033-4def-9ccd-39547f6602f4\") " Nov 01 20:21:47.644865 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:47.644797 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3118450b-b033-4def-9ccd-39547f6602f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3118450b-b033-4def-9ccd-39547f6602f4-istio-podinfo\") pod \"3118450b-b033-4def-9ccd-39547f6602f4\" (UID: \"3118450b-b033-4def-9ccd-39547f6602f4\") " Nov 01 20:21:47.644865 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:47.644841 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3118450b-b033-4def-9ccd-39547f6602f4-kube-api-access-4xp9x\" (UniqueName: \"kubernetes.io/projected/3118450b-b033-4def-9ccd-39547f6602f4-kube-api-access-4xp9x\") pod \"3118450b-b033-4def-9ccd-39547f6602f4\" (UID: \"3118450b-b033-4def-9ccd-39547f6602f4\") " Nov 01 20:21:47.644958 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:47.644898 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3118450b-b033-4def-9ccd-39547f6602f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3118450b-b033-4def-9ccd-39547f6602f4-istiod-ca-cert\") pod \"3118450b-b033-4def-9ccd-39547f6602f4\" (UID: \"3118450b-b033-4def-9ccd-39547f6602f4\") " Nov 01 20:21:47.644958 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:47.644837 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3118450b-b033-4def-9ccd-39547f6602f4-credential-socket" pod "3118450b-b033-4def-9ccd-39547f6602f4" (UID: "3118450b-b033-4def-9ccd-39547f6602f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:21:47.644958 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:47.644931 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3118450b-b033-4def-9ccd-39547f6602f4-istio-token\" (UniqueName: \"kubernetes.io/projected/3118450b-b033-4def-9ccd-39547f6602f4-istio-token\") pod \"3118450b-b033-4def-9ccd-39547f6602f4\" (UID: \"3118450b-b033-4def-9ccd-39547f6602f4\") " Nov 01 20:21:47.644958 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:47.644913 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3118450b-b033-4def-9ccd-39547f6602f4-istio-data" pod "3118450b-b033-4def-9ccd-39547f6602f4" (UID: "3118450b-b033-4def-9ccd-39547f6602f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:21:47.644958 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:47.644948 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3118450b-b033-4def-9ccd-39547f6602f4-workload-certs" pod "3118450b-b033-4def-9ccd-39547f6602f4" (UID: "3118450b-b033-4def-9ccd-39547f6602f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:21:47.645348 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:47.645277 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3118450b-b033-4def-9ccd-39547f6602f4-istiod-ca-cert" pod "3118450b-b033-4def-9ccd-39547f6602f4" (UID: "3118450b-b033-4def-9ccd-39547f6602f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:21:47.645592 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:47.645570 5935 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3118450b-b033-4def-9ccd-39547f6602f4-workload-certs\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:21:47.645681 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:47.645597 5935 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3118450b-b033-4def-9ccd-39547f6602f4-istiod-ca-cert\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:21:47.645681 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:47.645612 5935 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3118450b-b033-4def-9ccd-39547f6602f4-workload-socket\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:21:47.645681 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:47.645625 5935 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3118450b-b033-4def-9ccd-39547f6602f4-credential-socket\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:21:47.645681 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:47.645640 5935 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3118450b-b033-4def-9ccd-39547f6602f4-istio-data\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:21:47.646877 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:47.646854 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3118450b-b033-4def-9ccd-39547f6602f4-istio-envoy" pod "3118450b-b033-4def-9ccd-39547f6602f4" (UID: "3118450b-b033-4def-9ccd-39547f6602f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:21:47.647061 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:47.647044 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3118450b-b033-4def-9ccd-39547f6602f4-istio-token" pod "3118450b-b033-4def-9ccd-39547f6602f4" (UID: "3118450b-b033-4def-9ccd-39547f6602f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:21:47.647398 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:47.647366 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3118450b-b033-4def-9ccd-39547f6602f4-istio-podinfo" pod "3118450b-b033-4def-9ccd-39547f6602f4" (UID: "3118450b-b033-4def-9ccd-39547f6602f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:21:47.647398 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:47.647370 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3118450b-b033-4def-9ccd-39547f6602f4-kube-api-access-4xp9x" pod "3118450b-b033-4def-9ccd-39547f6602f4" (UID: "3118450b-b033-4def-9ccd-39547f6602f4"). InnerVolumeSpecName "kube-api-access-4xp9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:21:47.687175 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:47.687144 5935 generic.go:356] "Generic (PLEG): container finished" podID="3118450b-b033-4def-9ccd-39547f6602f4" containerID="2a8ea06c755e317176ba32e3fd0bb4c75fd4414b205f92f1642cad3f0d618348" exitCode=0 Nov 01 20:21:47.687354 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:47.687188 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w" Nov 01 20:21:47.687354 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:47.687183 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w" event={"ID":"3118450b-b033-4def-9ccd-39547f6602f4","Type":"ContainerDied","Data":"2a8ea06c755e317176ba32e3fd0bb4c75fd4414b205f92f1642cad3f0d618348"} Nov 01 20:21:47.687354 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:47.687281 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w" event={"ID":"3118450b-b033-4def-9ccd-39547f6602f4","Type":"ContainerDied","Data":"d45505a6e16292a4e669e905c74bdf5a2cbe9787e99a62b59721d5cbeddd50d1"} Nov 01 20:21:47.687354 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:47.687309 5935 scope.go:122] "RemoveContainer" containerID="2a8ea06c755e317176ba32e3fd0bb4c75fd4414b205f92f1642cad3f0d618348" Nov 01 20:21:47.689246 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:47.689217 5935 generic.go:356] "Generic (PLEG): container finished" podID="1fe8488d-838d-47fe-b2c5-8b88dfb5724e" containerID="122796766cd7ef50fe4d4c7a2ace79ee85d0bb38c329595b5333b9553653dd49" exitCode=0 Nov 01 20:21:47.689381 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:47.689303 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56" event={"ID":"1fe8488d-838d-47fe-b2c5-8b88dfb5724e","Type":"ContainerDied","Data":"122796766cd7ef50fe4d4c7a2ace79ee85d0bb38c329595b5333b9553653dd49"} Nov 01 20:21:47.698108 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:47.698089 5935 scope.go:122] "RemoveContainer" containerID="2a8ea06c755e317176ba32e3fd0bb4c75fd4414b205f92f1642cad3f0d618348" Nov 01 20:21:47.698373 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:21:47.698352 5935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a8ea06c755e317176ba32e3fd0bb4c75fd4414b205f92f1642cad3f0d618348\": container with ID starting with 2a8ea06c755e317176ba32e3fd0bb4c75fd4414b205f92f1642cad3f0d618348 not found: ID does not exist" containerID="2a8ea06c755e317176ba32e3fd0bb4c75fd4414b205f92f1642cad3f0d618348" Nov 01 20:21:47.698431 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:47.698383 5935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a8ea06c755e317176ba32e3fd0bb4c75fd4414b205f92f1642cad3f0d618348"} err="failed to get container status \"2a8ea06c755e317176ba32e3fd0bb4c75fd4414b205f92f1642cad3f0d618348\": rpc error: code = NotFound desc = could not find container \"2a8ea06c755e317176ba32e3fd0bb4c75fd4414b205f92f1642cad3f0d618348\": container with ID starting with 2a8ea06c755e317176ba32e3fd0bb4c75fd4414b205f92f1642cad3f0d618348 not found: ID does not exist" Nov 01 20:21:47.713536 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:47.713514 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w"] Nov 01 20:21:47.716402 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:47.716385 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-jt62w"] Nov 01 20:21:47.746951 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:47.746926 5935 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3118450b-b033-4def-9ccd-39547f6602f4-istio-podinfo\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:21:47.746951 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:47.746949 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xp9x\" (UniqueName: \"kubernetes.io/projected/3118450b-b033-4def-9ccd-39547f6602f4-kube-api-access-4xp9x\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:21:47.747095 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:47.746960 5935 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3118450b-b033-4def-9ccd-39547f6602f4-istio-token\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:21:47.747095 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:47.746969 5935 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3118450b-b033-4def-9ccd-39547f6602f4-istio-envoy\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:21:48.088986 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:48.088958 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56" Nov 01 20:21:48.151135 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:48.151102 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fe8488d-838d-47fe-b2c5-8b88dfb5724e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fe8488d-838d-47fe-b2c5-8b88dfb5724e-istiod-ca-cert\") pod \"1fe8488d-838d-47fe-b2c5-8b88dfb5724e\" (UID: \"1fe8488d-838d-47fe-b2c5-8b88dfb5724e\") " Nov 01 20:21:48.151135 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:48.151141 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fe8488d-838d-47fe-b2c5-8b88dfb5724e-istio-token\" (UniqueName: \"kubernetes.io/projected/1fe8488d-838d-47fe-b2c5-8b88dfb5724e-istio-token\") pod \"1fe8488d-838d-47fe-b2c5-8b88dfb5724e\" (UID: \"1fe8488d-838d-47fe-b2c5-8b88dfb5724e\") " Nov 01 20:21:48.151428 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:48.151234 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fe8488d-838d-47fe-b2c5-8b88dfb5724e-kube-api-access-885hx\" (UniqueName: \"kubernetes.io/projected/1fe8488d-838d-47fe-b2c5-8b88dfb5724e-kube-api-access-885hx\") pod \"1fe8488d-838d-47fe-b2c5-8b88dfb5724e\" (UID: \"1fe8488d-838d-47fe-b2c5-8b88dfb5724e\") " Nov 01 20:21:48.151428 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:48.151345 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fe8488d-838d-47fe-b2c5-8b88dfb5724e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fe8488d-838d-47fe-b2c5-8b88dfb5724e-workload-certs\") pod \"1fe8488d-838d-47fe-b2c5-8b88dfb5724e\" (UID: \"1fe8488d-838d-47fe-b2c5-8b88dfb5724e\") " Nov 01 20:21:48.151428 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:48.151413 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fe8488d-838d-47fe-b2c5-8b88dfb5724e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fe8488d-838d-47fe-b2c5-8b88dfb5724e-istio-data\") pod \"1fe8488d-838d-47fe-b2c5-8b88dfb5724e\" (UID: \"1fe8488d-838d-47fe-b2c5-8b88dfb5724e\") " Nov 01 20:21:48.151607 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:48.151458 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1fe8488d-838d-47fe-b2c5-8b88dfb5724e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fe8488d-838d-47fe-b2c5-8b88dfb5724e-istio-podinfo\") pod \"1fe8488d-838d-47fe-b2c5-8b88dfb5724e\" (UID: \"1fe8488d-838d-47fe-b2c5-8b88dfb5724e\") " Nov 01 20:21:48.151607 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:48.151559 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fe8488d-838d-47fe-b2c5-8b88dfb5724e-istiod-ca-cert" pod "1fe8488d-838d-47fe-b2c5-8b88dfb5724e" (UID: "1fe8488d-838d-47fe-b2c5-8b88dfb5724e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:21:48.151607 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:48.151580 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fe8488d-838d-47fe-b2c5-8b88dfb5724e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fe8488d-838d-47fe-b2c5-8b88dfb5724e-istio-envoy\") pod \"1fe8488d-838d-47fe-b2c5-8b88dfb5724e\" (UID: \"1fe8488d-838d-47fe-b2c5-8b88dfb5724e\") " Nov 01 20:21:48.151774 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:48.151675 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fe8488d-838d-47fe-b2c5-8b88dfb5724e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe8488d-838d-47fe-b2c5-8b88dfb5724e-credential-socket\") pod \"1fe8488d-838d-47fe-b2c5-8b88dfb5724e\" (UID: \"1fe8488d-838d-47fe-b2c5-8b88dfb5724e\") " Nov 01 20:21:48.151774 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:48.151721 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fe8488d-838d-47fe-b2c5-8b88dfb5724e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe8488d-838d-47fe-b2c5-8b88dfb5724e-workload-socket\") pod \"1fe8488d-838d-47fe-b2c5-8b88dfb5724e\" (UID: \"1fe8488d-838d-47fe-b2c5-8b88dfb5724e\") " Nov 01 20:21:48.151885 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:48.151815 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fe8488d-838d-47fe-b2c5-8b88dfb5724e-istio-data" pod "1fe8488d-838d-47fe-b2c5-8b88dfb5724e" (UID: "1fe8488d-838d-47fe-b2c5-8b88dfb5724e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:21:48.152268 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:48.152245 5935 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1fe8488d-838d-47fe-b2c5-8b88dfb5724e-istiod-ca-cert\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:21:48.152268 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:48.152269 5935 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1fe8488d-838d-47fe-b2c5-8b88dfb5724e-istio-data\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:21:48.152509 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:48.152351 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fe8488d-838d-47fe-b2c5-8b88dfb5724e-credential-socket" pod "1fe8488d-838d-47fe-b2c5-8b88dfb5724e" (UID: "1fe8488d-838d-47fe-b2c5-8b88dfb5724e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:21:48.152509 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:48.152366 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fe8488d-838d-47fe-b2c5-8b88dfb5724e-workload-socket" pod "1fe8488d-838d-47fe-b2c5-8b88dfb5724e" (UID: "1fe8488d-838d-47fe-b2c5-8b88dfb5724e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:21:48.152509 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:48.152414 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fe8488d-838d-47fe-b2c5-8b88dfb5724e-workload-certs" pod "1fe8488d-838d-47fe-b2c5-8b88dfb5724e" (UID: "1fe8488d-838d-47fe-b2c5-8b88dfb5724e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:21:48.154806 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:48.154779 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fe8488d-838d-47fe-b2c5-8b88dfb5724e-istio-envoy" pod "1fe8488d-838d-47fe-b2c5-8b88dfb5724e" (UID: "1fe8488d-838d-47fe-b2c5-8b88dfb5724e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:21:48.154961 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:48.154819 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1fe8488d-838d-47fe-b2c5-8b88dfb5724e-istio-podinfo" pod "1fe8488d-838d-47fe-b2c5-8b88dfb5724e" (UID: "1fe8488d-838d-47fe-b2c5-8b88dfb5724e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:21:48.155034 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:48.155006 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fe8488d-838d-47fe-b2c5-8b88dfb5724e-istio-token" pod "1fe8488d-838d-47fe-b2c5-8b88dfb5724e" (UID: "1fe8488d-838d-47fe-b2c5-8b88dfb5724e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:21:48.155140 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:48.155122 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fe8488d-838d-47fe-b2c5-8b88dfb5724e-kube-api-access-885hx" pod "1fe8488d-838d-47fe-b2c5-8b88dfb5724e" (UID: "1fe8488d-838d-47fe-b2c5-8b88dfb5724e"). InnerVolumeSpecName "kube-api-access-885hx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:21:48.253692 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:48.253663 5935 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1fe8488d-838d-47fe-b2c5-8b88dfb5724e-workload-certs\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:21:48.253692 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:48.253692 5935 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1fe8488d-838d-47fe-b2c5-8b88dfb5724e-istio-podinfo\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:21:48.253915 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:48.253702 5935 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1fe8488d-838d-47fe-b2c5-8b88dfb5724e-istio-envoy\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:21:48.253915 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:48.253711 5935 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe8488d-838d-47fe-b2c5-8b88dfb5724e-credential-socket\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:21:48.253915 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:48.253722 5935 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1fe8488d-838d-47fe-b2c5-8b88dfb5724e-workload-socket\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:21:48.253915 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:48.253730 5935 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1fe8488d-838d-47fe-b2c5-8b88dfb5724e-istio-token\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:21:48.253915 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:48.253739 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-885hx\" (UniqueName: \"kubernetes.io/projected/1fe8488d-838d-47fe-b2c5-8b88dfb5724e-kube-api-access-885hx\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:21:48.353865 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:48.353830 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3118450b-b033-4def-9ccd-39547f6602f4" path="/var/lib/kubelet/pods/3118450b-b033-4def-9ccd-39547f6602f4/volumes" Nov 01 20:21:48.701360 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:48.701323 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56" event={"ID":"1fe8488d-838d-47fe-b2c5-8b88dfb5724e","Type":"ContainerDied","Data":"c17e489a958b11ee57d45474d695617f9745bcba898aef4e69f4801cead71d4f"} Nov 01 20:21:48.701766 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:48.701393 5935 scope.go:122] "RemoveContainer" containerID="122796766cd7ef50fe4d4c7a2ace79ee85d0bb38c329595b5333b9553653dd49" Nov 01 20:21:48.701766 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:48.701338 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56" Nov 01 20:21:48.737433 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:48.737409 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56"] Nov 01 20:21:48.761778 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:48.761753 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-r5t56"] Nov 01 20:21:50.354783 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:50.354749 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fe8488d-838d-47fe-b2c5-8b88dfb5724e" path="/var/lib/kubelet/pods/1fe8488d-838d-47fe-b2c5-8b88dfb5724e/volumes" Nov 01 20:21:52.419449 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:52.419416 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-594czfxxq"] Nov 01 20:21:52.420187 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:52.420165 5935 cpu_manager.go:414] "containerMap: removing container" podUID="8f8d270f-189b-4434-b911-368d1082cb97" containerName="payload-processing" Nov 01 20:21:52.420277 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:52.420189 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f8d270f-189b-4434-b911-368d1082cb97" containerName="payload-processing" Nov 01 20:21:52.420277 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:52.420217 5935 cpu_manager.go:414] "containerMap: removing container" podUID="3118450b-b033-4def-9ccd-39547f6602f4" containerName="istio-proxy" Nov 01 20:21:52.420277 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:52.420225 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3118450b-b033-4def-9ccd-39547f6602f4" containerName="istio-proxy" Nov 01 20:21:52.420277 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:52.420249 5935 cpu_manager.go:414] "containerMap: removing container" podUID="1fe8488d-838d-47fe-b2c5-8b88dfb5724e" containerName="istio-proxy" Nov 01 20:21:52.420277 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:52.420256 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fe8488d-838d-47fe-b2c5-8b88dfb5724e" containerName="istio-proxy" Nov 01 20:21:52.420572 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:52.420413 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f8d270f-189b-4434-b911-368d1082cb97" containerName="payload-processing" Nov 01 20:21:52.420572 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:52.420429 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="3118450b-b033-4def-9ccd-39547f6602f4" containerName="istio-proxy" Nov 01 20:21:52.420572 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:52.420442 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fe8488d-838d-47fe-b2c5-8b88dfb5724e" containerName="istio-proxy" Nov 01 20:21:52.425545 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:52.425525 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-594czfxxq" Nov 01 20:21:52.434013 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:52.433792 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-aaea41\"/\"openshift-service-ca.crt\"" Nov 01 20:21:52.434013 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:52.433976 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-aaea41\"/\"e2e-cmp-legacy-d5c423-provider-cred\"" Nov 01 20:21:52.434013 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:52.433980 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-aaea41\"/\"payload-processing-external-model-e2e--245da986-dockercfg-nmptp\"" Nov 01 20:21:52.434190 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:52.434015 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-aaea41\"/\"routing-overlay\"" Nov 01 20:21:52.434190 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:52.434014 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-aaea41\"/\"payload-processing-external-model-plugins-e2e-extmod-a-aaea41\"" Nov 01 20:21:52.434190 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:52.433980 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-aaea41\"/\"kube-root-ca.crt\"" Nov 01 20:21:52.437484 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:52.437464 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-594czfxxq"] Nov 01 20:21:52.509200 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:52.509168 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/869bb0c3-ff27-456b-a778-b7c4a4e85be2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-594czfxxq\" (UID: \"869bb0c3-ff27-456b-a778-b7c4a4e85be2\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-594czfxxq" Nov 01 20:21:52.509404 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:52.509327 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxnqc\" (UniqueName: \"kubernetes.io/projected/869bb0c3-ff27-456b-a778-b7c4a4e85be2-kube-api-access-pxnqc\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-594czfxxq\" (UID: \"869bb0c3-ff27-456b-a778-b7c4a4e85be2\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-594czfxxq" Nov 01 20:21:52.509484 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:52.509470 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/869bb0c3-ff27-456b-a778-b7c4a4e85be2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-594czfxxq\" (UID: \"869bb0c3-ff27-456b-a778-b7c4a4e85be2\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-594czfxxq" Nov 01 20:21:52.509540 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:52.509526 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/869bb0c3-ff27-456b-a778-b7c4a4e85be2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-594czfxxq\" (UID: \"869bb0c3-ff27-456b-a778-b7c4a4e85be2\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-594czfxxq" Nov 01 20:21:52.619306 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:52.610977 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/869bb0c3-ff27-456b-a778-b7c4a4e85be2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-594czfxxq\" (UID: \"869bb0c3-ff27-456b-a778-b7c4a4e85be2\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-594czfxxq" Nov 01 20:21:52.619306 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:52.611088 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxnqc\" (UniqueName: \"kubernetes.io/projected/869bb0c3-ff27-456b-a778-b7c4a4e85be2-kube-api-access-pxnqc\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-594czfxxq\" (UID: \"869bb0c3-ff27-456b-a778-b7c4a4e85be2\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-594czfxxq" Nov 01 20:21:52.619306 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:52.611232 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/869bb0c3-ff27-456b-a778-b7c4a4e85be2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-594czfxxq\" (UID: \"869bb0c3-ff27-456b-a778-b7c4a4e85be2\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-594czfxxq" Nov 01 20:21:52.619306 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:52.611323 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/869bb0c3-ff27-456b-a778-b7c4a4e85be2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-594czfxxq\" (UID: \"869bb0c3-ff27-456b-a778-b7c4a4e85be2\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-594czfxxq" Nov 01 20:21:52.619306 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:52.612309 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/869bb0c3-ff27-456b-a778-b7c4a4e85be2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-594czfxxq\" (UID: \"869bb0c3-ff27-456b-a778-b7c4a4e85be2\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-594czfxxq" Nov 01 20:21:52.637620 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:52.637586 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/869bb0c3-ff27-456b-a778-b7c4a4e85be2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-594czfxxq\" (UID: \"869bb0c3-ff27-456b-a778-b7c4a4e85be2\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-594czfxxq" Nov 01 20:21:52.667803 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:52.667765 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxnqc\" (UniqueName: \"kubernetes.io/projected/869bb0c3-ff27-456b-a778-b7c4a4e85be2-kube-api-access-pxnqc\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-594czfxxq\" (UID: \"869bb0c3-ff27-456b-a778-b7c4a4e85be2\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-594czfxxq" Nov 01 20:21:52.668312 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:52.668257 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/869bb0c3-ff27-456b-a778-b7c4a4e85be2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-aaea41-594czfxxq\" (UID: \"869bb0c3-ff27-456b-a778-b7c4a4e85be2\") " pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-594czfxxq" Nov 01 20:21:52.741610 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:52.741572 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-594czfxxq" Nov 01 20:21:53.042445 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:53.042414 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-594czfxxq"] Nov 01 20:21:53.043159 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:21:53.043135 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod869bb0c3_ff27_456b_a778_b7c4a4e85be2.slice/crio-fe9c816aec38cf8e07ac38361076fdbb0e27c022959bdf4c55d954b0dad02fd0 WatchSource:0}: Error finding container fe9c816aec38cf8e07ac38361076fdbb0e27c022959bdf4c55d954b0dad02fd0: Status 404 returned error can't find the container with id fe9c816aec38cf8e07ac38361076fdbb0e27c022959bdf4c55d954b0dad02fd0 Nov 01 20:21:53.789274 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:53.789239 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-594czfxxq" event={"ID":"869bb0c3-ff27-456b-a778-b7c4a4e85be2","Type":"ContainerStarted","Data":"9231d343b174a0968f7cbb0705f698bb5f3fc99ffbbb6a69a0f0dac94b617c65"} Nov 01 20:21:53.789677 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:53.789302 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-594czfxxq" event={"ID":"869bb0c3-ff27-456b-a778-b7c4a4e85be2","Type":"ContainerStarted","Data":"fe9c816aec38cf8e07ac38361076fdbb0e27c022959bdf4c55d954b0dad02fd0"} Nov 01 20:21:53.789677 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:53.789397 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-594czfxxq" Nov 01 20:21:53.808808 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:53.808757 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-594czfxxq" podStartSLOduration=1.808743997 podStartE2EDuration="1.808743997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:21:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:21:53.805338649 +0000 UTC m=+1359.578499161" watchObservedRunningTime="2026-11-01 20:21:53.808743997 +0000 UTC m=+1359.581904510" Nov 01 20:21:55.412180 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:55.403657 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-933245-7cd656f7b9-7lszc"] Nov 01 20:21:55.412180 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:55.403949 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-933245-7cd656f7b9-7lszc" podUID="7f516592-de0d-441d-aa8a-85a22568a94b" containerName="maas-api" containerID="cri-o://fbd39275987feb181b5583a806732fab397cda040f38782352f5dd97a3d695ca" gracePeriod=30 Nov 01 20:21:55.834718 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:55.834473 5935 generic.go:356] "Generic (PLEG): container finished" podID="7f516592-de0d-441d-aa8a-85a22568a94b" containerID="fbd39275987feb181b5583a806732fab397cda040f38782352f5dd97a3d695ca" exitCode=0 Nov 01 20:21:55.834718 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:55.834630 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-933245-7cd656f7b9-7lszc" event={"ID":"7f516592-de0d-441d-aa8a-85a22568a94b","Type":"ContainerDied","Data":"fbd39275987feb181b5583a806732fab397cda040f38782352f5dd97a3d695ca"} Nov 01 20:21:55.871276 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:55.871251 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-933245-7cd656f7b9-7lszc" Nov 01 20:21:55.947941 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:55.947911 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f516592-de0d-441d-aa8a-85a22568a94b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f516592-de0d-441d-aa8a-85a22568a94b-openshift-service-ca\") pod \"7f516592-de0d-441d-aa8a-85a22568a94b\" (UID: \"7f516592-de0d-441d-aa8a-85a22568a94b\") " Nov 01 20:21:55.948123 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:55.948041 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f516592-de0d-441d-aa8a-85a22568a94b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f516592-de0d-441d-aa8a-85a22568a94b-metrics-certs\") pod \"7f516592-de0d-441d-aa8a-85a22568a94b\" (UID: \"7f516592-de0d-441d-aa8a-85a22568a94b\") " Nov 01 20:21:55.948171 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:55.948124 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f516592-de0d-441d-aa8a-85a22568a94b-kube-api-access-mgkfj\" (UniqueName: \"kubernetes.io/projected/7f516592-de0d-441d-aa8a-85a22568a94b-kube-api-access-mgkfj\") pod \"7f516592-de0d-441d-aa8a-85a22568a94b\" (UID: \"7f516592-de0d-441d-aa8a-85a22568a94b\") " Nov 01 20:21:55.948171 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:55.948156 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f516592-de0d-441d-aa8a-85a22568a94b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f516592-de0d-441d-aa8a-85a22568a94b-maas-api-tls\") pod \"7f516592-de0d-441d-aa8a-85a22568a94b\" (UID: \"7f516592-de0d-441d-aa8a-85a22568a94b\") " Nov 01 20:21:55.948277 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:55.948255 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f516592-de0d-441d-aa8a-85a22568a94b-openshift-service-ca" pod "7f516592-de0d-441d-aa8a-85a22568a94b" (UID: "7f516592-de0d-441d-aa8a-85a22568a94b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:21:55.948687 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:55.948663 5935 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f516592-de0d-441d-aa8a-85a22568a94b-openshift-service-ca\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:21:55.950411 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:55.950386 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f516592-de0d-441d-aa8a-85a22568a94b-metrics-certs" pod "7f516592-de0d-441d-aa8a-85a22568a94b" (UID: "7f516592-de0d-441d-aa8a-85a22568a94b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:21:55.950513 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:55.950408 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f516592-de0d-441d-aa8a-85a22568a94b-maas-api-tls" pod "7f516592-de0d-441d-aa8a-85a22568a94b" (UID: "7f516592-de0d-441d-aa8a-85a22568a94b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:21:55.950513 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:55.950493 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f516592-de0d-441d-aa8a-85a22568a94b-kube-api-access-mgkfj" pod "7f516592-de0d-441d-aa8a-85a22568a94b" (UID: "7f516592-de0d-441d-aa8a-85a22568a94b"). InnerVolumeSpecName "kube-api-access-mgkfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:21:56.050075 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:56.050042 5935 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f516592-de0d-441d-aa8a-85a22568a94b-metrics-certs\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:21:56.050075 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:56.050072 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgkfj\" (UniqueName: \"kubernetes.io/projected/7f516592-de0d-441d-aa8a-85a22568a94b-kube-api-access-mgkfj\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:21:56.050075 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:56.050083 5935 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f516592-de0d-441d-aa8a-85a22568a94b-maas-api-tls\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:21:56.842832 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:56.842800 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-933245-7cd656f7b9-7lszc" Nov 01 20:21:56.843319 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:56.842810 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-933245-7cd656f7b9-7lszc" event={"ID":"7f516592-de0d-441d-aa8a-85a22568a94b","Type":"ContainerDied","Data":"8d99b10dae328ef04e7a61f0b55162ce3387ace108b938b9367d8aca80b522e7"} Nov 01 20:21:56.843319 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:56.842933 5935 scope.go:122] "RemoveContainer" containerID="fbd39275987feb181b5583a806732fab397cda040f38782352f5dd97a3d695ca" Nov 01 20:21:56.862619 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:56.862590 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-933245-7cd656f7b9-7lszc"] Nov 01 20:21:56.868650 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:56.868627 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-933245-7cd656f7b9-7lszc"] Nov 01 20:21:58.358654 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:21:58.358613 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f516592-de0d-441d-aa8a-85a22568a94b" path="/var/lib/kubelet/pods/7f516592-de0d-441d-aa8a-85a22568a94b/volumes" Nov 01 20:22:02.828273 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:02.828234 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-57d281-55d489b495-lvrd9"] Nov 01 20:22:02.829023 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:02.829002 5935 cpu_manager.go:414] "containerMap: removing container" podUID="7f516592-de0d-441d-aa8a-85a22568a94b" containerName="maas-api" Nov 01 20:22:02.829119 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:02.829025 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f516592-de0d-441d-aa8a-85a22568a94b" containerName="maas-api" Nov 01 20:22:02.829197 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:02.829184 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f516592-de0d-441d-aa8a-85a22568a94b" containerName="maas-api" Nov 01 20:22:02.832498 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:02.832478 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-57d281-55d489b495-lvrd9" Nov 01 20:22:02.842147 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:02.841856 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-57d281\"" Nov 01 20:22:02.847047 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:02.847017 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-57d281-55d489b495-lvrd9"] Nov 01 20:22:02.853490 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:02.853226 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9e5d8fd-4547-409b-831a-ccfde839472c-plugins-config-volume\") pod \"payload-processing-e2e-explicit-57d281-55d489b495-lvrd9\" (UID: \"a9e5d8fd-4547-409b-831a-ccfde839472c\") " pod="openshift-ingress/payload-processing-e2e-explicit-57d281-55d489b495-lvrd9" Nov 01 20:22:02.853490 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:02.853342 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xbfr\" (UniqueName: \"kubernetes.io/projected/a9e5d8fd-4547-409b-831a-ccfde839472c-kube-api-access-8xbfr\") pod \"payload-processing-e2e-explicit-57d281-55d489b495-lvrd9\" (UID: \"a9e5d8fd-4547-409b-831a-ccfde839472c\") " pod="openshift-ingress/payload-processing-e2e-explicit-57d281-55d489b495-lvrd9" Nov 01 20:22:02.861177 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:02.861147 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-57d281-566d7fb6c-jbk6q"] Nov 01 20:22:02.876141 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:02.875807 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-57d281-566d7fb6c-jbk6q"] Nov 01 20:22:02.876141 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:02.875957 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-57d281-566d7fb6c-jbk6q" Nov 01 20:22:02.880983 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:02.880964 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-57d281-dockercfg-w52mz\"" Nov 01 20:22:02.963553 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:02.959811 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9e5d8fd-4547-409b-831a-ccfde839472c-plugins-config-volume\") pod \"payload-processing-e2e-explicit-57d281-55d489b495-lvrd9\" (UID: \"a9e5d8fd-4547-409b-831a-ccfde839472c\") " pod="openshift-ingress/payload-processing-e2e-explicit-57d281-55d489b495-lvrd9" Nov 01 20:22:02.965202 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:02.965174 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xbfr\" (UniqueName: \"kubernetes.io/projected/a9e5d8fd-4547-409b-831a-ccfde839472c-kube-api-access-8xbfr\") pod \"payload-processing-e2e-explicit-57d281-55d489b495-lvrd9\" (UID: \"a9e5d8fd-4547-409b-831a-ccfde839472c\") " pod="openshift-ingress/payload-processing-e2e-explicit-57d281-55d489b495-lvrd9" Nov 01 20:22:02.966449 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:02.966420 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9e5d8fd-4547-409b-831a-ccfde839472c-plugins-config-volume\") pod \"payload-processing-e2e-explicit-57d281-55d489b495-lvrd9\" (UID: \"a9e5d8fd-4547-409b-831a-ccfde839472c\") " pod="openshift-ingress/payload-processing-e2e-explicit-57d281-55d489b495-lvrd9" Nov 01 20:22:02.966586 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:02.966502 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc76fee9-e0bd-4ce6-9002-48182c70353c-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-57d281-566d7fb6c-jbk6q\" (UID: \"bc76fee9-e0bd-4ce6-9002-48182c70353c\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-57d281-566d7fb6c-jbk6q" Nov 01 20:22:02.966586 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:02.966555 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57vbs\" (UniqueName: \"kubernetes.io/projected/bc76fee9-e0bd-4ce6-9002-48182c70353c-kube-api-access-57vbs\") pod \"payload-pre-processing-e2e-explicit-57d281-566d7fb6c-jbk6q\" (UID: \"bc76fee9-e0bd-4ce6-9002-48182c70353c\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-57d281-566d7fb6c-jbk6q" Nov 01 20:22:03.002221 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:03.002146 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xbfr\" (UniqueName: \"kubernetes.io/projected/a9e5d8fd-4547-409b-831a-ccfde839472c-kube-api-access-8xbfr\") pod \"payload-processing-e2e-explicit-57d281-55d489b495-lvrd9\" (UID: \"a9e5d8fd-4547-409b-831a-ccfde839472c\") " pod="openshift-ingress/payload-processing-e2e-explicit-57d281-55d489b495-lvrd9" Nov 01 20:22:03.068377 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:03.068333 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc76fee9-e0bd-4ce6-9002-48182c70353c-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-57d281-566d7fb6c-jbk6q\" (UID: \"bc76fee9-e0bd-4ce6-9002-48182c70353c\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-57d281-566d7fb6c-jbk6q" Nov 01 20:22:03.068641 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:03.068619 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57vbs\" (UniqueName: \"kubernetes.io/projected/bc76fee9-e0bd-4ce6-9002-48182c70353c-kube-api-access-57vbs\") pod \"payload-pre-processing-e2e-explicit-57d281-566d7fb6c-jbk6q\" (UID: \"bc76fee9-e0bd-4ce6-9002-48182c70353c\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-57d281-566d7fb6c-jbk6q" Nov 01 20:22:03.069937 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:03.069911 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc76fee9-e0bd-4ce6-9002-48182c70353c-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-57d281-566d7fb6c-jbk6q\" (UID: \"bc76fee9-e0bd-4ce6-9002-48182c70353c\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-57d281-566d7fb6c-jbk6q" Nov 01 20:22:03.076667 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:03.076644 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57vbs\" (UniqueName: \"kubernetes.io/projected/bc76fee9-e0bd-4ce6-9002-48182c70353c-kube-api-access-57vbs\") pod \"payload-pre-processing-e2e-explicit-57d281-566d7fb6c-jbk6q\" (UID: \"bc76fee9-e0bd-4ce6-9002-48182c70353c\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-57d281-566d7fb6c-jbk6q" Nov 01 20:22:03.155698 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:03.155610 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-57d281-55d489b495-lvrd9" Nov 01 20:22:03.199122 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:03.199087 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-57d281-566d7fb6c-jbk6q" Nov 01 20:22:03.447938 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:03.447902 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-57d281-55d489b495-lvrd9"] Nov 01 20:22:03.448241 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:22:03.448216 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9e5d8fd_4547_409b_831a_ccfde839472c.slice/crio-f1c84e9352fd7d5c33243e55cf8a66bef8e257ca3afcc84dda87ffb373a0c2e2 WatchSource:0}: Error finding container f1c84e9352fd7d5c33243e55cf8a66bef8e257ca3afcc84dda87ffb373a0c2e2: Status 404 returned error can't find the container with id f1c84e9352fd7d5c33243e55cf8a66bef8e257ca3afcc84dda87ffb373a0c2e2 Nov 01 20:22:03.492492 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:03.492464 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-57d281-566d7fb6c-jbk6q"] Nov 01 20:22:03.904541 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:03.904451 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-57d281-566d7fb6c-jbk6q" event={"ID":"bc76fee9-e0bd-4ce6-9002-48182c70353c","Type":"ContainerStarted","Data":"329fa3fdaeadc4ee62df3aa9ae90236f700a63d93b074bfc70b2f01af008f99b"} Nov 01 20:22:03.904541 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:03.904518 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-57d281-566d7fb6c-jbk6q" Nov 01 20:22:03.905030 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:03.904542 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-57d281-566d7fb6c-jbk6q" event={"ID":"bc76fee9-e0bd-4ce6-9002-48182c70353c","Type":"ContainerStarted","Data":"ed38269da61e53fb36c06ffb5e1eedb226a4bec5297a077e15258e57020b70f2"} Nov 01 20:22:03.906350 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:03.906318 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-57d281-55d489b495-lvrd9" event={"ID":"a9e5d8fd-4547-409b-831a-ccfde839472c","Type":"ContainerStarted","Data":"646f333b57730c9e09def9e0a2bf98799cd3f78928d13c53683fa2456fc25346"} Nov 01 20:22:03.906506 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:03.906368 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-57d281-55d489b495-lvrd9" event={"ID":"a9e5d8fd-4547-409b-831a-ccfde839472c","Type":"ContainerStarted","Data":"f1c84e9352fd7d5c33243e55cf8a66bef8e257ca3afcc84dda87ffb373a0c2e2"} Nov 01 20:22:03.906506 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:03.906388 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-57d281-55d489b495-lvrd9" Nov 01 20:22:03.920941 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:03.920890 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-57d281-566d7fb6c-jbk6q" podStartSLOduration=1.920876845 podStartE2EDuration="1.920876845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:22:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:22:03.919313852 +0000 UTC m=+1369.692474371" watchObservedRunningTime="2026-11-01 20:22:03.920876845 +0000 UTC m=+1369.694037357" Nov 01 20:22:03.936731 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:03.936689 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-57d281-55d489b495-lvrd9" podStartSLOduration=1.936676788 podStartE2EDuration="1.936676788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:22:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:22:03.93394188 +0000 UTC m=+1369.707102386" watchObservedRunningTime="2026-11-01 20:22:03.936676788 +0000 UTC m=+1369.709837312" Nov 01 20:22:04.798724 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:04.798698 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-594czfxxq" Nov 01 20:22:05.845717 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:22:05.845677 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:22:14.915376 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:14.915347 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-57d281-55d489b495-lvrd9" Nov 01 20:22:14.915881 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:14.915411 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-57d281-566d7fb6c-jbk6q" Nov 01 20:22:28.252580 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:28.252540 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-57d281/explicit-model-57d281-kserve-65f56fc8b9-sqs5z"] Nov 01 20:22:28.261211 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:28.261173 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-57d281/explicit-model-57d281-kserve-65f56fc8b9-sqs5z" Nov 01 20:22:28.264553 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:28.264500 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-57d281/explicit-model-57d281-kserve-65f56fc8b9-sqs5z"] Nov 01 20:22:28.268246 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:28.268225 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-57d281\"/\"explicit-model-57d281-kserve-self-signed-certs\"" Nov 01 20:22:28.270989 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:28.270967 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-57d281\"/\"openshift-service-ca.crt\"" Nov 01 20:22:28.271196 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:28.271178 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-57d281\"/\"default-dockercfg-mhth8\"" Nov 01 20:22:28.271479 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:28.271460 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-57d281\"/\"kube-root-ca.crt\"" Nov 01 20:22:28.315239 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:28.315210 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a5dbfdf-8d49-46a4-b271-0688f3ec3606-tls-certs\") pod \"explicit-model-57d281-kserve-65f56fc8b9-sqs5z\" (UID: \"8a5dbfdf-8d49-46a4-b271-0688f3ec3606\") " pod="ai-tenant-e2e-explicit-57d281/explicit-model-57d281-kserve-65f56fc8b9-sqs5z" Nov 01 20:22:28.315413 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:28.315315 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a5dbfdf-8d49-46a4-b271-0688f3ec3606-home\") pod \"explicit-model-57d281-kserve-65f56fc8b9-sqs5z\" (UID: \"8a5dbfdf-8d49-46a4-b271-0688f3ec3606\") " pod="ai-tenant-e2e-explicit-57d281/explicit-model-57d281-kserve-65f56fc8b9-sqs5z" Nov 01 20:22:28.315467 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:28.315437 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a5dbfdf-8d49-46a4-b271-0688f3ec3606-model-cache\") pod \"explicit-model-57d281-kserve-65f56fc8b9-sqs5z\" (UID: \"8a5dbfdf-8d49-46a4-b271-0688f3ec3606\") " pod="ai-tenant-e2e-explicit-57d281/explicit-model-57d281-kserve-65f56fc8b9-sqs5z" Nov 01 20:22:28.315518 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:28.315480 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a5dbfdf-8d49-46a4-b271-0688f3ec3606-dshm\") pod \"explicit-model-57d281-kserve-65f56fc8b9-sqs5z\" (UID: \"8a5dbfdf-8d49-46a4-b271-0688f3ec3606\") " pod="ai-tenant-e2e-explicit-57d281/explicit-model-57d281-kserve-65f56fc8b9-sqs5z" Nov 01 20:22:28.315587 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:28.315562 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgczq\" (UniqueName: \"kubernetes.io/projected/8a5dbfdf-8d49-46a4-b271-0688f3ec3606-kube-api-access-kgczq\") pod \"explicit-model-57d281-kserve-65f56fc8b9-sqs5z\" (UID: \"8a5dbfdf-8d49-46a4-b271-0688f3ec3606\") " pod="ai-tenant-e2e-explicit-57d281/explicit-model-57d281-kserve-65f56fc8b9-sqs5z" Nov 01 20:22:28.315704 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:28.315685 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a5dbfdf-8d49-46a4-b271-0688f3ec3606-tmp-dir\") pod \"explicit-model-57d281-kserve-65f56fc8b9-sqs5z\" (UID: \"8a5dbfdf-8d49-46a4-b271-0688f3ec3606\") " pod="ai-tenant-e2e-explicit-57d281/explicit-model-57d281-kserve-65f56fc8b9-sqs5z" Nov 01 20:22:28.417574 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:28.417533 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a5dbfdf-8d49-46a4-b271-0688f3ec3606-tls-certs\") pod \"explicit-model-57d281-kserve-65f56fc8b9-sqs5z\" (UID: \"8a5dbfdf-8d49-46a4-b271-0688f3ec3606\") " pod="ai-tenant-e2e-explicit-57d281/explicit-model-57d281-kserve-65f56fc8b9-sqs5z" Nov 01 20:22:28.417772 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:28.417614 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a5dbfdf-8d49-46a4-b271-0688f3ec3606-home\") pod \"explicit-model-57d281-kserve-65f56fc8b9-sqs5z\" (UID: \"8a5dbfdf-8d49-46a4-b271-0688f3ec3606\") " pod="ai-tenant-e2e-explicit-57d281/explicit-model-57d281-kserve-65f56fc8b9-sqs5z" Nov 01 20:22:28.417772 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:28.417684 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a5dbfdf-8d49-46a4-b271-0688f3ec3606-model-cache\") pod \"explicit-model-57d281-kserve-65f56fc8b9-sqs5z\" (UID: \"8a5dbfdf-8d49-46a4-b271-0688f3ec3606\") " pod="ai-tenant-e2e-explicit-57d281/explicit-model-57d281-kserve-65f56fc8b9-sqs5z" Nov 01 20:22:28.417772 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:28.417717 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a5dbfdf-8d49-46a4-b271-0688f3ec3606-dshm\") pod \"explicit-model-57d281-kserve-65f56fc8b9-sqs5z\" (UID: \"8a5dbfdf-8d49-46a4-b271-0688f3ec3606\") " pod="ai-tenant-e2e-explicit-57d281/explicit-model-57d281-kserve-65f56fc8b9-sqs5z" Nov 01 20:22:28.417772 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:28.417745 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgczq\" (UniqueName: \"kubernetes.io/projected/8a5dbfdf-8d49-46a4-b271-0688f3ec3606-kube-api-access-kgczq\") pod \"explicit-model-57d281-kserve-65f56fc8b9-sqs5z\" (UID: \"8a5dbfdf-8d49-46a4-b271-0688f3ec3606\") " pod="ai-tenant-e2e-explicit-57d281/explicit-model-57d281-kserve-65f56fc8b9-sqs5z" Nov 01 20:22:28.418000 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:28.417840 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a5dbfdf-8d49-46a4-b271-0688f3ec3606-tmp-dir\") pod \"explicit-model-57d281-kserve-65f56fc8b9-sqs5z\" (UID: \"8a5dbfdf-8d49-46a4-b271-0688f3ec3606\") " pod="ai-tenant-e2e-explicit-57d281/explicit-model-57d281-kserve-65f56fc8b9-sqs5z" Nov 01 20:22:28.418413 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:28.418385 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a5dbfdf-8d49-46a4-b271-0688f3ec3606-tmp-dir\") pod \"explicit-model-57d281-kserve-65f56fc8b9-sqs5z\" (UID: \"8a5dbfdf-8d49-46a4-b271-0688f3ec3606\") " pod="ai-tenant-e2e-explicit-57d281/explicit-model-57d281-kserve-65f56fc8b9-sqs5z" Nov 01 20:22:28.418765 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:28.418739 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a5dbfdf-8d49-46a4-b271-0688f3ec3606-model-cache\") pod \"explicit-model-57d281-kserve-65f56fc8b9-sqs5z\" (UID: \"8a5dbfdf-8d49-46a4-b271-0688f3ec3606\") " pod="ai-tenant-e2e-explicit-57d281/explicit-model-57d281-kserve-65f56fc8b9-sqs5z" Nov 01 20:22:28.419004 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:28.418985 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a5dbfdf-8d49-46a4-b271-0688f3ec3606-home\") pod \"explicit-model-57d281-kserve-65f56fc8b9-sqs5z\" (UID: \"8a5dbfdf-8d49-46a4-b271-0688f3ec3606\") " pod="ai-tenant-e2e-explicit-57d281/explicit-model-57d281-kserve-65f56fc8b9-sqs5z" Nov 01 20:22:28.434353 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:28.434302 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a5dbfdf-8d49-46a4-b271-0688f3ec3606-tls-certs\") pod \"explicit-model-57d281-kserve-65f56fc8b9-sqs5z\" (UID: \"8a5dbfdf-8d49-46a4-b271-0688f3ec3606\") " pod="ai-tenant-e2e-explicit-57d281/explicit-model-57d281-kserve-65f56fc8b9-sqs5z" Nov 01 20:22:28.434505 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:28.434494 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a5dbfdf-8d49-46a4-b271-0688f3ec3606-dshm\") pod \"explicit-model-57d281-kserve-65f56fc8b9-sqs5z\" (UID: \"8a5dbfdf-8d49-46a4-b271-0688f3ec3606\") " pod="ai-tenant-e2e-explicit-57d281/explicit-model-57d281-kserve-65f56fc8b9-sqs5z" Nov 01 20:22:28.438528 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:28.438448 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgczq\" (UniqueName: \"kubernetes.io/projected/8a5dbfdf-8d49-46a4-b271-0688f3ec3606-kube-api-access-kgczq\") pod \"explicit-model-57d281-kserve-65f56fc8b9-sqs5z\" (UID: \"8a5dbfdf-8d49-46a4-b271-0688f3ec3606\") " pod="ai-tenant-e2e-explicit-57d281/explicit-model-57d281-kserve-65f56fc8b9-sqs5z" Nov 01 20:22:28.588786 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:28.588700 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-57d281/explicit-model-57d281-kserve-65f56fc8b9-sqs5z" Nov 01 20:22:28.869199 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:28.869173 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-57d281/explicit-model-57d281-kserve-65f56fc8b9-sqs5z"] Nov 01 20:22:28.871231 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:22:28.871189 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a5dbfdf_8d49_46a4_b271_0688f3ec3606.slice/crio-c6fc7617f2ab6021686edf61daecad532945b05638e4e9dd719e74098eceb78b WatchSource:0}: Error finding container c6fc7617f2ab6021686edf61daecad532945b05638e4e9dd719e74098eceb78b: Status 404 returned error can't find the container with id c6fc7617f2ab6021686edf61daecad532945b05638e4e9dd719e74098eceb78b Nov 01 20:22:29.131341 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:29.131229 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-57d281/explicit-model-57d281-kserve-65f56fc8b9-sqs5z" event={"ID":"8a5dbfdf-8d49-46a4-b271-0688f3ec3606","Type":"ContainerStarted","Data":"db7edd596b948e2f36919c21b0ffd76827160c0fc676daa9ec42898c403233a0"} Nov 01 20:22:29.131341 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:29.131279 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-57d281/explicit-model-57d281-kserve-65f56fc8b9-sqs5z" event={"ID":"8a5dbfdf-8d49-46a4-b271-0688f3ec3606","Type":"ContainerStarted","Data":"c6fc7617f2ab6021686edf61daecad532945b05638e4e9dd719e74098eceb78b"} Nov 01 20:22:29.153978 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:29.153920 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-57d281/explicit-model-57d281-kserve-65f56fc8b9-sqs5z" podStartSLOduration=1.153901043 podStartE2EDuration="1.153901043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:22:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:22:29.150695936 +0000 UTC m=+1394.923856453" watchObservedRunningTime="2026-11-01 20:22:29.153901043 +0000 UTC m=+1394.927061559" Nov 01 20:22:29.589001 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:29.588969 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-57d281/explicit-model-57d281-kserve-65f56fc8b9-sqs5z" Nov 01 20:22:35.155355 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:35.155324 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-57d281/explicit-model-57d281-kserve-65f56fc8b9-sqs5z"] Nov 01 20:22:35.847546 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:22:35.847513 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:22:42.511855 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:42.511820 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k"] Nov 01 20:22:42.512277 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:42.512037 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k" podUID="7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a" containerName="istio-proxy" containerID="cri-o://c7b40909a9d6f1c85c274b821cd3a83987377e042d04e96d01bd1e7f89ba2305" gracePeriod=30 Nov 01 20:22:47.771353 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:47.771327 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k" Nov 01 20:22:47.892599 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:47.892525 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a-kube-api-access-lg4ks\" (UniqueName: \"kubernetes.io/projected/7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a-kube-api-access-lg4ks\") pod \"7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a\" (UID: \"7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a\") " Nov 01 20:22:47.892763 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:47.892611 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a-workload-certs\") pod \"7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a\" (UID: \"7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a\") " Nov 01 20:22:47.892763 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:47.892644 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a-workload-socket\") pod \"7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a\" (UID: \"7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a\") " Nov 01 20:22:47.892763 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:47.892711 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a-istio-podinfo\") pod \"7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a\" (UID: \"7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a\") " Nov 01 20:22:47.892763 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:47.892731 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a-istio-envoy\") pod \"7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a\" (UID: \"7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a\") " Nov 01 20:22:47.892763 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:47.892755 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a-istio-token\" (UniqueName: \"kubernetes.io/projected/7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a-istio-token\") pod \"7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a\" (UID: \"7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a\") " Nov 01 20:22:47.893101 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:47.892812 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a-credential-socket\") pod \"7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a\" (UID: \"7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a\") " Nov 01 20:22:47.893101 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:47.892909 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a-istiod-ca-cert\") pod \"7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a\" (UID: \"7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a\") " Nov 01 20:22:47.893101 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:47.892944 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a-istio-data\") pod \"7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a\" (UID: \"7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a\") " Nov 01 20:22:47.893101 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:47.892946 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a-workload-certs" pod "7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a" (UID: "7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:22:47.893360 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:47.893109 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a-workload-socket" pod "7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a" (UID: "7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:22:47.893360 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:47.893193 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a-credential-socket" pod "7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a" (UID: "7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:22:47.893360 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:47.893328 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a-istiod-ca-cert" pod "7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a" (UID: "7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:22:47.893519 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:47.893370 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a-istio-data" pod "7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a" (UID: "7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:22:47.893573 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:47.893545 5935 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a-credential-socket\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:22:47.893573 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:47.893565 5935 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a-istiod-ca-cert\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:22:47.893667 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:47.893581 5935 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a-istio-data\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:22:47.893667 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:47.893595 5935 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a-workload-certs\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:22:47.893667 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:47.893610 5935 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a-workload-socket\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:22:47.894821 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:47.894799 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a-kube-api-access-lg4ks" pod "7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a" (UID: "7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a"). InnerVolumeSpecName "kube-api-access-lg4ks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:22:47.894934 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:47.894838 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a-istio-token" pod "7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a" (UID: "7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:22:47.895344 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:47.895320 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a-istio-envoy" pod "7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a" (UID: "7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:22:47.895459 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:47.895442 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a-istio-podinfo" pod "7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a" (UID: "7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:22:47.994770 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:47.994742 5935 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a-istio-podinfo\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:22:47.994770 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:47.994768 5935 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a-istio-envoy\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:22:47.994770 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:47.994777 5935 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a-istio-token\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:22:47.994989 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:47.994787 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lg4ks\" (UniqueName: \"kubernetes.io/projected/7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a-kube-api-access-lg4ks\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:22:48.282650 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:48.282611 5935 generic.go:356] "Generic (PLEG): container finished" podID="7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a" containerID="c7b40909a9d6f1c85c274b821cd3a83987377e042d04e96d01bd1e7f89ba2305" exitCode=0 Nov 01 20:22:48.282811 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:48.282651 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k" Nov 01 20:22:48.282811 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:48.282660 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k" event={"ID":"7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a","Type":"ContainerDied","Data":"c7b40909a9d6f1c85c274b821cd3a83987377e042d04e96d01bd1e7f89ba2305"} Nov 01 20:22:48.282811 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:48.282715 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k" event={"ID":"7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a","Type":"ContainerDied","Data":"995c5edb50fd55762876dd7222f422c9ae45cc9ed83091c080d8469cea3eb7f8"} Nov 01 20:22:48.282811 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:48.282742 5935 scope.go:122] "RemoveContainer" containerID="c7b40909a9d6f1c85c274b821cd3a83987377e042d04e96d01bd1e7f89ba2305" Nov 01 20:22:48.294196 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:48.294181 5935 scope.go:122] "RemoveContainer" containerID="c7b40909a9d6f1c85c274b821cd3a83987377e042d04e96d01bd1e7f89ba2305" Nov 01 20:22:48.294505 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:22:48.294483 5935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7b40909a9d6f1c85c274b821cd3a83987377e042d04e96d01bd1e7f89ba2305\": container with ID starting with c7b40909a9d6f1c85c274b821cd3a83987377e042d04e96d01bd1e7f89ba2305 not found: ID does not exist" containerID="c7b40909a9d6f1c85c274b821cd3a83987377e042d04e96d01bd1e7f89ba2305" Nov 01 20:22:48.294556 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:48.294514 5935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7b40909a9d6f1c85c274b821cd3a83987377e042d04e96d01bd1e7f89ba2305"} err="failed to get container status \"c7b40909a9d6f1c85c274b821cd3a83987377e042d04e96d01bd1e7f89ba2305\": rpc error: code = NotFound desc = could not find container \"c7b40909a9d6f1c85c274b821cd3a83987377e042d04e96d01bd1e7f89ba2305\": container with ID starting with c7b40909a9d6f1c85c274b821cd3a83987377e042d04e96d01bd1e7f89ba2305 not found: ID does not exist" Nov 01 20:22:48.307614 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:48.307586 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k"] Nov 01 20:22:48.311076 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:48.311056 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-bgt6k"] Nov 01 20:22:48.354163 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:48.354137 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a" path="/var/lib/kubelet/pods/7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a/volumes" Nov 01 20:22:50.186509 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:50.186472 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-57d281/explicit-model-57d281-kserve-65f56fc8b9-sqs5z" podUID="8a5dbfdf-8d49-46a4-b271-0688f3ec3606" containerName="main" containerID="cri-o://db7edd596b948e2f36919c21b0ffd76827160c0fc676daa9ec42898c403233a0" gracePeriod=45 Nov 01 20:22:50.456448 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:50.456421 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-57d281/explicit-model-57d281-kserve-65f56fc8b9-sqs5z" Nov 01 20:22:50.525893 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:50.525855 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a5dbfdf-8d49-46a4-b271-0688f3ec3606-home\" (UniqueName: \"kubernetes.io/empty-dir/8a5dbfdf-8d49-46a4-b271-0688f3ec3606-home\") pod \"8a5dbfdf-8d49-46a4-b271-0688f3ec3606\" (UID: \"8a5dbfdf-8d49-46a4-b271-0688f3ec3606\") " Nov 01 20:22:50.525893 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:50.525895 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a5dbfdf-8d49-46a4-b271-0688f3ec3606-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a5dbfdf-8d49-46a4-b271-0688f3ec3606-model-cache\") pod \"8a5dbfdf-8d49-46a4-b271-0688f3ec3606\" (UID: \"8a5dbfdf-8d49-46a4-b271-0688f3ec3606\") " Nov 01 20:22:50.526126 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:50.525981 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a5dbfdf-8d49-46a4-b271-0688f3ec3606-tls-certs\" (UniqueName: \"kubernetes.io/secret/8a5dbfdf-8d49-46a4-b271-0688f3ec3606-tls-certs\") pod \"8a5dbfdf-8d49-46a4-b271-0688f3ec3606\" (UID: \"8a5dbfdf-8d49-46a4-b271-0688f3ec3606\") " Nov 01 20:22:50.526126 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:50.526007 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a5dbfdf-8d49-46a4-b271-0688f3ec3606-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a5dbfdf-8d49-46a4-b271-0688f3ec3606-dshm\") pod \"8a5dbfdf-8d49-46a4-b271-0688f3ec3606\" (UID: \"8a5dbfdf-8d49-46a4-b271-0688f3ec3606\") " Nov 01 20:22:50.526232 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:50.526137 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a5dbfdf-8d49-46a4-b271-0688f3ec3606-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a5dbfdf-8d49-46a4-b271-0688f3ec3606-tmp-dir\") pod \"8a5dbfdf-8d49-46a4-b271-0688f3ec3606\" (UID: \"8a5dbfdf-8d49-46a4-b271-0688f3ec3606\") " Nov 01 20:22:50.526232 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:50.526176 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a5dbfdf-8d49-46a4-b271-0688f3ec3606-home" pod "8a5dbfdf-8d49-46a4-b271-0688f3ec3606" (UID: "8a5dbfdf-8d49-46a4-b271-0688f3ec3606"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:22:50.526373 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:50.526227 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a5dbfdf-8d49-46a4-b271-0688f3ec3606-kube-api-access-kgczq\" (UniqueName: \"kubernetes.io/projected/8a5dbfdf-8d49-46a4-b271-0688f3ec3606-kube-api-access-kgczq\") pod \"8a5dbfdf-8d49-46a4-b271-0688f3ec3606\" (UID: \"8a5dbfdf-8d49-46a4-b271-0688f3ec3606\") " Nov 01 20:22:50.526373 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:50.526249 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a5dbfdf-8d49-46a4-b271-0688f3ec3606-model-cache" pod "8a5dbfdf-8d49-46a4-b271-0688f3ec3606" (UID: "8a5dbfdf-8d49-46a4-b271-0688f3ec3606"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:22:50.526489 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:50.526403 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a5dbfdf-8d49-46a4-b271-0688f3ec3606-tmp-dir" pod "8a5dbfdf-8d49-46a4-b271-0688f3ec3606" (UID: "8a5dbfdf-8d49-46a4-b271-0688f3ec3606"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:22:50.526805 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:50.526782 5935 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8a5dbfdf-8d49-46a4-b271-0688f3ec3606-home\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:22:50.526805 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:50.526807 5935 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8a5dbfdf-8d49-46a4-b271-0688f3ec3606-model-cache\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:22:50.526976 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:50.526817 5935 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8a5dbfdf-8d49-46a4-b271-0688f3ec3606-tmp-dir\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:22:50.528420 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:50.528392 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a5dbfdf-8d49-46a4-b271-0688f3ec3606-kube-api-access-kgczq" pod "8a5dbfdf-8d49-46a4-b271-0688f3ec3606" (UID: "8a5dbfdf-8d49-46a4-b271-0688f3ec3606"). InnerVolumeSpecName "kube-api-access-kgczq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:22:50.528572 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:50.528552 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a5dbfdf-8d49-46a4-b271-0688f3ec3606-dshm" pod "8a5dbfdf-8d49-46a4-b271-0688f3ec3606" (UID: "8a5dbfdf-8d49-46a4-b271-0688f3ec3606"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:22:50.528620 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:50.528551 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a5dbfdf-8d49-46a4-b271-0688f3ec3606-tls-certs" pod "8a5dbfdf-8d49-46a4-b271-0688f3ec3606" (UID: "8a5dbfdf-8d49-46a4-b271-0688f3ec3606"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:22:50.627685 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:50.627650 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgczq\" (UniqueName: \"kubernetes.io/projected/8a5dbfdf-8d49-46a4-b271-0688f3ec3606-kube-api-access-kgczq\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:22:50.627685 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:50.627682 5935 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8a5dbfdf-8d49-46a4-b271-0688f3ec3606-tls-certs\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:22:50.627901 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:50.627696 5935 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8a5dbfdf-8d49-46a4-b271-0688f3ec3606-dshm\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:22:51.307455 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:51.307418 5935 generic.go:356] "Generic (PLEG): container finished" podID="8a5dbfdf-8d49-46a4-b271-0688f3ec3606" containerID="db7edd596b948e2f36919c21b0ffd76827160c0fc676daa9ec42898c403233a0" exitCode=0 Nov 01 20:22:51.307949 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:51.307472 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-57d281/explicit-model-57d281-kserve-65f56fc8b9-sqs5z" Nov 01 20:22:51.307949 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:51.307519 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-57d281/explicit-model-57d281-kserve-65f56fc8b9-sqs5z" event={"ID":"8a5dbfdf-8d49-46a4-b271-0688f3ec3606","Type":"ContainerDied","Data":"db7edd596b948e2f36919c21b0ffd76827160c0fc676daa9ec42898c403233a0"} Nov 01 20:22:51.307949 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:51.307563 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-57d281/explicit-model-57d281-kserve-65f56fc8b9-sqs5z" event={"ID":"8a5dbfdf-8d49-46a4-b271-0688f3ec3606","Type":"ContainerDied","Data":"c6fc7617f2ab6021686edf61daecad532945b05638e4e9dd719e74098eceb78b"} Nov 01 20:22:51.307949 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:51.307584 5935 scope.go:122] "RemoveContainer" containerID="db7edd596b948e2f36919c21b0ffd76827160c0fc676daa9ec42898c403233a0" Nov 01 20:22:51.322787 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:51.322761 5935 scope.go:122] "RemoveContainer" containerID="db7edd596b948e2f36919c21b0ffd76827160c0fc676daa9ec42898c403233a0" Nov 01 20:22:51.323122 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:22:51.323098 5935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db7edd596b948e2f36919c21b0ffd76827160c0fc676daa9ec42898c403233a0\": container with ID starting with db7edd596b948e2f36919c21b0ffd76827160c0fc676daa9ec42898c403233a0 not found: ID does not exist" containerID="db7edd596b948e2f36919c21b0ffd76827160c0fc676daa9ec42898c403233a0" Nov 01 20:22:51.323219 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:51.323132 5935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db7edd596b948e2f36919c21b0ffd76827160c0fc676daa9ec42898c403233a0"} err="failed to get container status \"db7edd596b948e2f36919c21b0ffd76827160c0fc676daa9ec42898c403233a0\": rpc error: code = NotFound desc = could not find container \"db7edd596b948e2f36919c21b0ffd76827160c0fc676daa9ec42898c403233a0\": container with ID starting with db7edd596b948e2f36919c21b0ffd76827160c0fc676daa9ec42898c403233a0 not found: ID does not exist" Nov 01 20:22:51.336280 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:51.336250 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-57d281/explicit-model-57d281-kserve-65f56fc8b9-sqs5z"] Nov 01 20:22:51.340803 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:51.340780 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-57d281/explicit-model-57d281-kserve-65f56fc8b9-sqs5z"] Nov 01 20:22:51.948988 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:51.948943 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2babaef6-77b4b5bd9b-k7zzz"] Nov 01 20:22:51.949913 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:51.949887 5935 cpu_manager.go:414] "containerMap: removing container" podUID="8a5dbfdf-8d49-46a4-b271-0688f3ec3606" containerName="main" Nov 01 20:22:51.949913 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:51.949912 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a5dbfdf-8d49-46a4-b271-0688f3ec3606" containerName="main" Nov 01 20:22:51.950117 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:51.949929 5935 cpu_manager.go:414] "containerMap: removing container" podUID="7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a" containerName="istio-proxy" Nov 01 20:22:51.950117 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:51.949938 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a" containerName="istio-proxy" Nov 01 20:22:51.950223 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:51.950153 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a5dbfdf-8d49-46a4-b271-0688f3ec3606" containerName="main" Nov 01 20:22:51.950223 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:51.950169 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f4577ff-f9ab-4b7b-b1a3-3d66f31cdb7a" containerName="istio-proxy" Nov 01 20:22:51.955723 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:51.955700 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2babaef6-77b4b5bd9b-k7zzz" Nov 01 20:22:51.961891 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:51.961873 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-2babaef6-serving-cert\"" Nov 01 20:22:51.963581 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:51.963560 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2babaef6-77b4b5bd9b-k7zzz"] Nov 01 20:22:52.044858 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:52.044830 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49e78298-4681-4a39-89b3-65eeae2d3c22-maas-api-tls\") pod \"maas-api-e2e-mt-2babaef6-77b4b5bd9b-k7zzz\" (UID: \"49e78298-4681-4a39-89b3-65eeae2d3c22\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2babaef6-77b4b5bd9b-k7zzz" Nov 01 20:22:52.045019 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:52.044900 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49e78298-4681-4a39-89b3-65eeae2d3c22-openshift-service-ca\") pod \"maas-api-e2e-mt-2babaef6-77b4b5bd9b-k7zzz\" (UID: \"49e78298-4681-4a39-89b3-65eeae2d3c22\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2babaef6-77b4b5bd9b-k7zzz" Nov 01 20:22:52.045063 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:52.045045 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49e78298-4681-4a39-89b3-65eeae2d3c22-metrics-certs\") pod \"maas-api-e2e-mt-2babaef6-77b4b5bd9b-k7zzz\" (UID: \"49e78298-4681-4a39-89b3-65eeae2d3c22\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2babaef6-77b4b5bd9b-k7zzz" Nov 01 20:22:52.045145 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:52.045096 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9s7v\" (UniqueName: \"kubernetes.io/projected/49e78298-4681-4a39-89b3-65eeae2d3c22-kube-api-access-t9s7v\") pod \"maas-api-e2e-mt-2babaef6-77b4b5bd9b-k7zzz\" (UID: \"49e78298-4681-4a39-89b3-65eeae2d3c22\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2babaef6-77b4b5bd9b-k7zzz" Nov 01 20:22:52.146318 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:52.146251 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49e78298-4681-4a39-89b3-65eeae2d3c22-maas-api-tls\") pod \"maas-api-e2e-mt-2babaef6-77b4b5bd9b-k7zzz\" (UID: \"49e78298-4681-4a39-89b3-65eeae2d3c22\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2babaef6-77b4b5bd9b-k7zzz" Nov 01 20:22:52.146490 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:52.146352 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49e78298-4681-4a39-89b3-65eeae2d3c22-openshift-service-ca\") pod \"maas-api-e2e-mt-2babaef6-77b4b5bd9b-k7zzz\" (UID: \"49e78298-4681-4a39-89b3-65eeae2d3c22\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2babaef6-77b4b5bd9b-k7zzz" Nov 01 20:22:52.146490 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:52.146449 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49e78298-4681-4a39-89b3-65eeae2d3c22-metrics-certs\") pod \"maas-api-e2e-mt-2babaef6-77b4b5bd9b-k7zzz\" (UID: \"49e78298-4681-4a39-89b3-65eeae2d3c22\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2babaef6-77b4b5bd9b-k7zzz" Nov 01 20:22:52.146490 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:52.146472 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9s7v\" (UniqueName: \"kubernetes.io/projected/49e78298-4681-4a39-89b3-65eeae2d3c22-kube-api-access-t9s7v\") pod \"maas-api-e2e-mt-2babaef6-77b4b5bd9b-k7zzz\" (UID: \"49e78298-4681-4a39-89b3-65eeae2d3c22\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2babaef6-77b4b5bd9b-k7zzz" Nov 01 20:22:52.147692 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:52.147661 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49e78298-4681-4a39-89b3-65eeae2d3c22-openshift-service-ca\") pod \"maas-api-e2e-mt-2babaef6-77b4b5bd9b-k7zzz\" (UID: \"49e78298-4681-4a39-89b3-65eeae2d3c22\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2babaef6-77b4b5bd9b-k7zzz" Nov 01 20:22:52.150416 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:52.150389 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49e78298-4681-4a39-89b3-65eeae2d3c22-maas-api-tls\") pod \"maas-api-e2e-mt-2babaef6-77b4b5bd9b-k7zzz\" (UID: \"49e78298-4681-4a39-89b3-65eeae2d3c22\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2babaef6-77b4b5bd9b-k7zzz" Nov 01 20:22:52.152309 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:52.152263 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49e78298-4681-4a39-89b3-65eeae2d3c22-metrics-certs\") pod \"maas-api-e2e-mt-2babaef6-77b4b5bd9b-k7zzz\" (UID: \"49e78298-4681-4a39-89b3-65eeae2d3c22\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2babaef6-77b4b5bd9b-k7zzz" Nov 01 20:22:52.154798 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:52.154771 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9s7v\" (UniqueName: \"kubernetes.io/projected/49e78298-4681-4a39-89b3-65eeae2d3c22-kube-api-access-t9s7v\") pod \"maas-api-e2e-mt-2babaef6-77b4b5bd9b-k7zzz\" (UID: \"49e78298-4681-4a39-89b3-65eeae2d3c22\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2babaef6-77b4b5bd9b-k7zzz" Nov 01 20:22:52.272852 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:52.272774 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2babaef6-77b4b5bd9b-k7zzz" Nov 01 20:22:52.364855 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:52.364820 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a5dbfdf-8d49-46a4-b271-0688f3ec3606" path="/var/lib/kubelet/pods/8a5dbfdf-8d49-46a4-b271-0688f3ec3606/volumes" Nov 01 20:22:52.553853 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:52.553805 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2babaef6-77b4b5bd9b-k7zzz"] Nov 01 20:22:53.324065 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:53.323986 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2babaef6-77b4b5bd9b-k7zzz" event={"ID":"49e78298-4681-4a39-89b3-65eeae2d3c22","Type":"ContainerStarted","Data":"14d9fd31b38825bfc26c680c2dee59d4d9c2efc1fb78caf281b11192d7ec174d"} Nov 01 20:22:53.324065 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:53.324043 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2babaef6-77b4b5bd9b-k7zzz" Nov 01 20:22:53.324065 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:53.324060 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2babaef6-77b4b5bd9b-k7zzz" event={"ID":"49e78298-4681-4a39-89b3-65eeae2d3c22","Type":"ContainerStarted","Data":"fa6c2969274c61dc51b9a0c8129778bfc0d2381ac029da0853abe95bffd9592f"} Nov 01 20:22:53.343793 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:53.343747 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2babaef6-77b4b5bd9b-k7zzz" podStartSLOduration=1.959923861 podStartE2EDuration="2.343734162s" totalImagesPullingTime="383.810301ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:22:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:22:53.339709636 +0000 UTC m=+1419.112870148" watchObservedRunningTime="2026-11-01 20:22:53.343734162 +0000 UTC m=+1419.116894674" Nov 01 20:22:59.336265 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:22:59.336238 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2babaef6-77b4b5bd9b-k7zzz" Nov 01 20:23:05.844960 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:23:05.844921 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:23:11.788365 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:11.788328 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-2babaef6/e2e-lifecycle-model-2babaef6-kserve-c7444bf-twlvv"] Nov 01 20:23:11.793580 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:11.793555 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2babaef6/e2e-lifecycle-model-2babaef6-kserve-c7444bf-twlvv" Nov 01 20:23:11.799244 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:11.799221 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-2babaef6\"/\"openshift-service-ca.crt\"" Nov 01 20:23:11.800858 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:11.800821 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-2babaef6\"/\"kube-root-ca.crt\"" Nov 01 20:23:11.801538 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:11.801522 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-2babaef6\"/\"default-dockercfg-4mcrf\"" Nov 01 20:23:11.801720 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:11.801697 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-2babaef6\"/\"e2e-lifecycle-model-2babaef6-kserve-self-signed-certs\"" Nov 01 20:23:11.802918 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:11.802891 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-2babaef6/e2e-lifecycle-model-2babaef6-kserve-c7444bf-twlvv"] Nov 01 20:23:11.924536 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:11.924503 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/098cc87f-f041-4d32-b909-be5a16ced4d8-home\") pod \"e2e-lifecycle-model-2babaef6-kserve-c7444bf-twlvv\" (UID: \"098cc87f-f041-4d32-b909-be5a16ced4d8\") " pod="ai-tenant-e2e-mt-2babaef6/e2e-lifecycle-model-2babaef6-kserve-c7444bf-twlvv" Nov 01 20:23:11.924735 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:11.924643 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/098cc87f-f041-4d32-b909-be5a16ced4d8-tmp-dir\") pod \"e2e-lifecycle-model-2babaef6-kserve-c7444bf-twlvv\" (UID: \"098cc87f-f041-4d32-b909-be5a16ced4d8\") " pod="ai-tenant-e2e-mt-2babaef6/e2e-lifecycle-model-2babaef6-kserve-c7444bf-twlvv" Nov 01 20:23:11.924810 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:11.924772 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/098cc87f-f041-4d32-b909-be5a16ced4d8-model-cache\") pod \"e2e-lifecycle-model-2babaef6-kserve-c7444bf-twlvv\" (UID: \"098cc87f-f041-4d32-b909-be5a16ced4d8\") " pod="ai-tenant-e2e-mt-2babaef6/e2e-lifecycle-model-2babaef6-kserve-c7444bf-twlvv" Nov 01 20:23:11.924890 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:11.924815 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/098cc87f-f041-4d32-b909-be5a16ced4d8-tls-certs\") pod \"e2e-lifecycle-model-2babaef6-kserve-c7444bf-twlvv\" (UID: \"098cc87f-f041-4d32-b909-be5a16ced4d8\") " pod="ai-tenant-e2e-mt-2babaef6/e2e-lifecycle-model-2babaef6-kserve-c7444bf-twlvv" Nov 01 20:23:11.924890 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:11.924877 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/098cc87f-f041-4d32-b909-be5a16ced4d8-dshm\") pod \"e2e-lifecycle-model-2babaef6-kserve-c7444bf-twlvv\" (UID: \"098cc87f-f041-4d32-b909-be5a16ced4d8\") " pod="ai-tenant-e2e-mt-2babaef6/e2e-lifecycle-model-2babaef6-kserve-c7444bf-twlvv" Nov 01 20:23:11.925084 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:11.925059 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbp8n\" (UniqueName: \"kubernetes.io/projected/098cc87f-f041-4d32-b909-be5a16ced4d8-kube-api-access-pbp8n\") pod \"e2e-lifecycle-model-2babaef6-kserve-c7444bf-twlvv\" (UID: \"098cc87f-f041-4d32-b909-be5a16ced4d8\") " pod="ai-tenant-e2e-mt-2babaef6/e2e-lifecycle-model-2babaef6-kserve-c7444bf-twlvv" Nov 01 20:23:12.026139 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:12.026100 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/098cc87f-f041-4d32-b909-be5a16ced4d8-model-cache\") pod \"e2e-lifecycle-model-2babaef6-kserve-c7444bf-twlvv\" (UID: \"098cc87f-f041-4d32-b909-be5a16ced4d8\") " pod="ai-tenant-e2e-mt-2babaef6/e2e-lifecycle-model-2babaef6-kserve-c7444bf-twlvv" Nov 01 20:23:12.026327 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:12.026153 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/098cc87f-f041-4d32-b909-be5a16ced4d8-tls-certs\") pod \"e2e-lifecycle-model-2babaef6-kserve-c7444bf-twlvv\" (UID: \"098cc87f-f041-4d32-b909-be5a16ced4d8\") " pod="ai-tenant-e2e-mt-2babaef6/e2e-lifecycle-model-2babaef6-kserve-c7444bf-twlvv" Nov 01 20:23:12.026327 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:12.026182 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/098cc87f-f041-4d32-b909-be5a16ced4d8-dshm\") pod \"e2e-lifecycle-model-2babaef6-kserve-c7444bf-twlvv\" (UID: \"098cc87f-f041-4d32-b909-be5a16ced4d8\") " pod="ai-tenant-e2e-mt-2babaef6/e2e-lifecycle-model-2babaef6-kserve-c7444bf-twlvv" Nov 01 20:23:12.026426 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:12.026357 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbp8n\" (UniqueName: \"kubernetes.io/projected/098cc87f-f041-4d32-b909-be5a16ced4d8-kube-api-access-pbp8n\") pod \"e2e-lifecycle-model-2babaef6-kserve-c7444bf-twlvv\" (UID: \"098cc87f-f041-4d32-b909-be5a16ced4d8\") " pod="ai-tenant-e2e-mt-2babaef6/e2e-lifecycle-model-2babaef6-kserve-c7444bf-twlvv" Nov 01 20:23:12.026464 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:12.026446 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/098cc87f-f041-4d32-b909-be5a16ced4d8-home\") pod \"e2e-lifecycle-model-2babaef6-kserve-c7444bf-twlvv\" (UID: \"098cc87f-f041-4d32-b909-be5a16ced4d8\") " pod="ai-tenant-e2e-mt-2babaef6/e2e-lifecycle-model-2babaef6-kserve-c7444bf-twlvv" Nov 01 20:23:12.026532 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:12.026517 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/098cc87f-f041-4d32-b909-be5a16ced4d8-tmp-dir\") pod \"e2e-lifecycle-model-2babaef6-kserve-c7444bf-twlvv\" (UID: \"098cc87f-f041-4d32-b909-be5a16ced4d8\") " pod="ai-tenant-e2e-mt-2babaef6/e2e-lifecycle-model-2babaef6-kserve-c7444bf-twlvv" Nov 01 20:23:12.027036 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:12.027006 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/098cc87f-f041-4d32-b909-be5a16ced4d8-tmp-dir\") pod \"e2e-lifecycle-model-2babaef6-kserve-c7444bf-twlvv\" (UID: \"098cc87f-f041-4d32-b909-be5a16ced4d8\") " pod="ai-tenant-e2e-mt-2babaef6/e2e-lifecycle-model-2babaef6-kserve-c7444bf-twlvv" Nov 01 20:23:12.027319 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:12.027279 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/098cc87f-f041-4d32-b909-be5a16ced4d8-model-cache\") pod \"e2e-lifecycle-model-2babaef6-kserve-c7444bf-twlvv\" (UID: \"098cc87f-f041-4d32-b909-be5a16ced4d8\") " pod="ai-tenant-e2e-mt-2babaef6/e2e-lifecycle-model-2babaef6-kserve-c7444bf-twlvv" Nov 01 20:23:12.028166 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:12.028134 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/098cc87f-f041-4d32-b909-be5a16ced4d8-home\") pod \"e2e-lifecycle-model-2babaef6-kserve-c7444bf-twlvv\" (UID: \"098cc87f-f041-4d32-b909-be5a16ced4d8\") " pod="ai-tenant-e2e-mt-2babaef6/e2e-lifecycle-model-2babaef6-kserve-c7444bf-twlvv" Nov 01 20:23:12.031152 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:12.031126 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/098cc87f-f041-4d32-b909-be5a16ced4d8-tls-certs\") pod \"e2e-lifecycle-model-2babaef6-kserve-c7444bf-twlvv\" (UID: \"098cc87f-f041-4d32-b909-be5a16ced4d8\") " pod="ai-tenant-e2e-mt-2babaef6/e2e-lifecycle-model-2babaef6-kserve-c7444bf-twlvv" Nov 01 20:23:12.032808 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:12.032504 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/098cc87f-f041-4d32-b909-be5a16ced4d8-dshm\") pod \"e2e-lifecycle-model-2babaef6-kserve-c7444bf-twlvv\" (UID: \"098cc87f-f041-4d32-b909-be5a16ced4d8\") " pod="ai-tenant-e2e-mt-2babaef6/e2e-lifecycle-model-2babaef6-kserve-c7444bf-twlvv" Nov 01 20:23:12.036384 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:12.036358 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbp8n\" (UniqueName: \"kubernetes.io/projected/098cc87f-f041-4d32-b909-be5a16ced4d8-kube-api-access-pbp8n\") pod \"e2e-lifecycle-model-2babaef6-kserve-c7444bf-twlvv\" (UID: \"098cc87f-f041-4d32-b909-be5a16ced4d8\") " pod="ai-tenant-e2e-mt-2babaef6/e2e-lifecycle-model-2babaef6-kserve-c7444bf-twlvv" Nov 01 20:23:12.113715 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:12.113166 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2babaef6/e2e-lifecycle-model-2babaef6-kserve-c7444bf-twlvv" Nov 01 20:23:12.395057 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:12.395024 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-2babaef6/e2e-lifecycle-model-2babaef6-kserve-c7444bf-twlvv"] Nov 01 20:23:12.396967 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:23:12.396942 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod098cc87f_f041_4d32_b909_be5a16ced4d8.slice/crio-0482572908e62aca02d2cbca0dd7343b4ece63c077d5cf3295d6094d25cc4d9b WatchSource:0}: Error finding container 0482572908e62aca02d2cbca0dd7343b4ece63c077d5cf3295d6094d25cc4d9b: Status 404 returned error can't find the container with id 0482572908e62aca02d2cbca0dd7343b4ece63c077d5cf3295d6094d25cc4d9b Nov 01 20:23:12.463423 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:12.463388 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2babaef6/e2e-lifecycle-model-2babaef6-kserve-c7444bf-twlvv" event={"ID":"098cc87f-f041-4d32-b909-be5a16ced4d8","Type":"ContainerStarted","Data":"0482572908e62aca02d2cbca0dd7343b4ece63c077d5cf3295d6094d25cc4d9b"} Nov 01 20:23:13.471920 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:13.471884 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2babaef6/e2e-lifecycle-model-2babaef6-kserve-c7444bf-twlvv" event={"ID":"098cc87f-f041-4d32-b909-be5a16ced4d8","Type":"ContainerStarted","Data":"c3219d8fa524b7a216f1d9b201635e74a23ee0ff42917441074b35874257e44c"} Nov 01 20:23:13.489659 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:13.489604 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-2babaef6/e2e-lifecycle-model-2babaef6-kserve-c7444bf-twlvv" podStartSLOduration=2.489591765 podStartE2EDuration="2.489591765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:23:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:23:13.487723026 +0000 UTC m=+1439.260883539" watchObservedRunningTime="2026-11-01 20:23:13.489591765 +0000 UTC m=+1439.262752316" Nov 01 20:23:14.134320 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:14.122392 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-2babaef6/e2e-lifecycle-model-2babaef6-kserve-c7444bf-twlvv" Nov 01 20:23:15.185958 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:15.185850 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-594czfxxq"] Nov 01 20:23:16.726505 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:16.726469 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-57d281-55d489b495-lvrd9"] Nov 01 20:23:16.738919 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:16.738885 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-57d281-566d7fb6c-jbk6q"] Nov 01 20:23:20.186840 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:20.186748 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-594czfxxq" podUID="869bb0c3-ff27-456b-a778-b7c4a4e85be2" containerName="payload-processing" containerID="cri-o://9231d343b174a0968f7cbb0705f698bb5f3fc99ffbbb6a69a0f0dac94b617c65" gracePeriod=25 Nov 01 20:23:20.476164 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:20.476136 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-594czfxxq" Nov 01 20:23:20.520466 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:20.520425 5935 generic.go:356] "Generic (PLEG): container finished" podID="869bb0c3-ff27-456b-a778-b7c4a4e85be2" containerID="9231d343b174a0968f7cbb0705f698bb5f3fc99ffbbb6a69a0f0dac94b617c65" exitCode=0 Nov 01 20:23:20.520620 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:20.520474 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-594czfxxq" event={"ID":"869bb0c3-ff27-456b-a778-b7c4a4e85be2","Type":"ContainerDied","Data":"9231d343b174a0968f7cbb0705f698bb5f3fc99ffbbb6a69a0f0dac94b617c65"} Nov 01 20:23:20.520620 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:20.520487 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-594czfxxq" Nov 01 20:23:20.520620 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:20.520531 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-594czfxxq" event={"ID":"869bb0c3-ff27-456b-a778-b7c4a4e85be2","Type":"ContainerDied","Data":"fe9c816aec38cf8e07ac38361076fdbb0e27c022959bdf4c55d954b0dad02fd0"} Nov 01 20:23:20.520620 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:20.520564 5935 scope.go:122] "RemoveContainer" containerID="9231d343b174a0968f7cbb0705f698bb5f3fc99ffbbb6a69a0f0dac94b617c65" Nov 01 20:23:20.532260 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:20.532243 5935 scope.go:122] "RemoveContainer" containerID="9231d343b174a0968f7cbb0705f698bb5f3fc99ffbbb6a69a0f0dac94b617c65" Nov 01 20:23:20.532606 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:23:20.532568 5935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9231d343b174a0968f7cbb0705f698bb5f3fc99ffbbb6a69a0f0dac94b617c65\": container with ID starting with 9231d343b174a0968f7cbb0705f698bb5f3fc99ffbbb6a69a0f0dac94b617c65 not found: ID does not exist" containerID="9231d343b174a0968f7cbb0705f698bb5f3fc99ffbbb6a69a0f0dac94b617c65" Nov 01 20:23:20.532677 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:20.532619 5935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9231d343b174a0968f7cbb0705f698bb5f3fc99ffbbb6a69a0f0dac94b617c65"} err="failed to get container status \"9231d343b174a0968f7cbb0705f698bb5f3fc99ffbbb6a69a0f0dac94b617c65\": rpc error: code = NotFound desc = could not find container \"9231d343b174a0968f7cbb0705f698bb5f3fc99ffbbb6a69a0f0dac94b617c65\": container with ID starting with 9231d343b174a0968f7cbb0705f698bb5f3fc99ffbbb6a69a0f0dac94b617c65 not found: ID does not exist" Nov 01 20:23:20.563231 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:20.563210 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/869bb0c3-ff27-456b-a778-b7c4a4e85be2-kube-api-access-pxnqc\" (UniqueName: \"kubernetes.io/projected/869bb0c3-ff27-456b-a778-b7c4a4e85be2-kube-api-access-pxnqc\") pod \"869bb0c3-ff27-456b-a778-b7c4a4e85be2\" (UID: \"869bb0c3-ff27-456b-a778-b7c4a4e85be2\") " Nov 01 20:23:20.563338 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:20.563325 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/869bb0c3-ff27-456b-a778-b7c4a4e85be2-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/869bb0c3-ff27-456b-a778-b7c4a4e85be2-routing-overlay-volume\") pod \"869bb0c3-ff27-456b-a778-b7c4a4e85be2\" (UID: \"869bb0c3-ff27-456b-a778-b7c4a4e85be2\") " Nov 01 20:23:20.563384 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:20.563358 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/869bb0c3-ff27-456b-a778-b7c4a4e85be2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/869bb0c3-ff27-456b-a778-b7c4a4e85be2-plugins-config-volume\") pod \"869bb0c3-ff27-456b-a778-b7c4a4e85be2\" (UID: \"869bb0c3-ff27-456b-a778-b7c4a4e85be2\") " Nov 01 20:23:20.563475 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:20.563456 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/869bb0c3-ff27-456b-a778-b7c4a4e85be2-provider-credentials\" (UniqueName: \"kubernetes.io/projected/869bb0c3-ff27-456b-a778-b7c4a4e85be2-provider-credentials\") pod \"869bb0c3-ff27-456b-a778-b7c4a4e85be2\" (UID: \"869bb0c3-ff27-456b-a778-b7c4a4e85be2\") " Nov 01 20:23:20.563726 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:20.563684 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/869bb0c3-ff27-456b-a778-b7c4a4e85be2-routing-overlay-volume" pod "869bb0c3-ff27-456b-a778-b7c4a4e85be2" (UID: "869bb0c3-ff27-456b-a778-b7c4a4e85be2"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:23:20.563819 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:20.563775 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/869bb0c3-ff27-456b-a778-b7c4a4e85be2-plugins-config-volume" pod "869bb0c3-ff27-456b-a778-b7c4a4e85be2" (UID: "869bb0c3-ff27-456b-a778-b7c4a4e85be2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:23:20.564007 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:20.563985 5935 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/869bb0c3-ff27-456b-a778-b7c4a4e85be2-routing-overlay-volume\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:23:20.564054 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:20.564014 5935 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/869bb0c3-ff27-456b-a778-b7c4a4e85be2-plugins-config-volume\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:23:20.565352 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:20.565331 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/869bb0c3-ff27-456b-a778-b7c4a4e85be2-kube-api-access-pxnqc" pod "869bb0c3-ff27-456b-a778-b7c4a4e85be2" (UID: "869bb0c3-ff27-456b-a778-b7c4a4e85be2"). InnerVolumeSpecName "kube-api-access-pxnqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:23:20.565529 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:20.565508 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/869bb0c3-ff27-456b-a778-b7c4a4e85be2-provider-credentials" pod "869bb0c3-ff27-456b-a778-b7c4a4e85be2" (UID: "869bb0c3-ff27-456b-a778-b7c4a4e85be2"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:23:20.665754 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:20.665717 5935 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/869bb0c3-ff27-456b-a778-b7c4a4e85be2-provider-credentials\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:23:20.665754 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:20.665749 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxnqc\" (UniqueName: \"kubernetes.io/projected/869bb0c3-ff27-456b-a778-b7c4a4e85be2-kube-api-access-pxnqc\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:23:20.854387 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:20.854354 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-594czfxxq"] Nov 01 20:23:20.863273 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:20.863247 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-aaea41/payload-processing-external-model-e2e-extmod-a-aaea41-594czfxxq"] Nov 01 20:23:21.736398 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:21.727659 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-57d281-55d489b495-lvrd9" podUID="a9e5d8fd-4547-409b-831a-ccfde839472c" containerName="payload-processing" containerID="cri-o://646f333b57730c9e09def9e0a2bf98799cd3f78928d13c53683fa2456fc25346" gracePeriod=25 Nov 01 20:23:21.745088 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:21.742461 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-57d281-566d7fb6c-jbk6q" podUID="bc76fee9-e0bd-4ce6-9002-48182c70353c" containerName="payload-pre-processing" containerID="cri-o://329fa3fdaeadc4ee62df3aa9ae90236f700a63d93b074bfc70b2f01af008f99b" gracePeriod=25 Nov 01 20:23:22.021141 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:22.021119 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-57d281-55d489b495-lvrd9" Nov 01 20:23:22.055621 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:22.055595 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-57d281-566d7fb6c-jbk6q" Nov 01 20:23:22.086866 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:22.086838 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc76fee9-e0bd-4ce6-9002-48182c70353c-kube-api-access-57vbs\" (UniqueName: \"kubernetes.io/projected/bc76fee9-e0bd-4ce6-9002-48182c70353c-kube-api-access-57vbs\") pod \"bc76fee9-e0bd-4ce6-9002-48182c70353c\" (UID: \"bc76fee9-e0bd-4ce6-9002-48182c70353c\") " Nov 01 20:23:22.086995 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:22.086891 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9e5d8fd-4547-409b-831a-ccfde839472c-kube-api-access-8xbfr\" (UniqueName: \"kubernetes.io/projected/a9e5d8fd-4547-409b-831a-ccfde839472c-kube-api-access-8xbfr\") pod \"a9e5d8fd-4547-409b-831a-ccfde839472c\" (UID: \"a9e5d8fd-4547-409b-831a-ccfde839472c\") " Nov 01 20:23:22.087090 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:22.087078 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9e5d8fd-4547-409b-831a-ccfde839472c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9e5d8fd-4547-409b-831a-ccfde839472c-plugins-config-volume\") pod \"a9e5d8fd-4547-409b-831a-ccfde839472c\" (UID: \"a9e5d8fd-4547-409b-831a-ccfde839472c\") " Nov 01 20:23:22.087143 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:22.087132 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc76fee9-e0bd-4ce6-9002-48182c70353c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc76fee9-e0bd-4ce6-9002-48182c70353c-plugins-config-volume\") pod \"bc76fee9-e0bd-4ce6-9002-48182c70353c\" (UID: \"bc76fee9-e0bd-4ce6-9002-48182c70353c\") " Nov 01 20:23:22.087500 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:22.087471 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9e5d8fd-4547-409b-831a-ccfde839472c-plugins-config-volume" pod "a9e5d8fd-4547-409b-831a-ccfde839472c" (UID: "a9e5d8fd-4547-409b-831a-ccfde839472c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:23:22.087601 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:22.087504 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc76fee9-e0bd-4ce6-9002-48182c70353c-plugins-config-volume" pod "bc76fee9-e0bd-4ce6-9002-48182c70353c" (UID: "bc76fee9-e0bd-4ce6-9002-48182c70353c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:23:22.087706 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:22.087687 5935 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9e5d8fd-4547-409b-831a-ccfde839472c-plugins-config-volume\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:23:22.087759 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:22.087711 5935 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc76fee9-e0bd-4ce6-9002-48182c70353c-plugins-config-volume\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:23:22.089037 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:22.089015 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc76fee9-e0bd-4ce6-9002-48182c70353c-kube-api-access-57vbs" pod "bc76fee9-e0bd-4ce6-9002-48182c70353c" (UID: "bc76fee9-e0bd-4ce6-9002-48182c70353c"). InnerVolumeSpecName "kube-api-access-57vbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:23:22.089093 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:22.089054 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9e5d8fd-4547-409b-831a-ccfde839472c-kube-api-access-8xbfr" pod "a9e5d8fd-4547-409b-831a-ccfde839472c" (UID: "a9e5d8fd-4547-409b-831a-ccfde839472c"). InnerVolumeSpecName "kube-api-access-8xbfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:23:22.114345 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:22.114322 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-2babaef6/e2e-lifecycle-model-2babaef6-kserve-c7444bf-twlvv" Nov 01 20:23:22.126977 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:22.126957 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-2babaef6/e2e-lifecycle-model-2babaef6-kserve-c7444bf-twlvv" Nov 01 20:23:22.188853 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:22.188823 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xbfr\" (UniqueName: \"kubernetes.io/projected/a9e5d8fd-4547-409b-831a-ccfde839472c-kube-api-access-8xbfr\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:23:22.188853 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:22.188847 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57vbs\" (UniqueName: \"kubernetes.io/projected/bc76fee9-e0bd-4ce6-9002-48182c70353c-kube-api-access-57vbs\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:23:22.354381 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:22.354312 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="869bb0c3-ff27-456b-a778-b7c4a4e85be2" path="/var/lib/kubelet/pods/869bb0c3-ff27-456b-a778-b7c4a4e85be2/volumes" Nov 01 20:23:22.536930 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:22.536893 5935 generic.go:356] "Generic (PLEG): container finished" podID="bc76fee9-e0bd-4ce6-9002-48182c70353c" containerID="329fa3fdaeadc4ee62df3aa9ae90236f700a63d93b074bfc70b2f01af008f99b" exitCode=0 Nov 01 20:23:22.537098 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:22.536943 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-57d281-566d7fb6c-jbk6q" Nov 01 20:23:22.537098 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:22.536972 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-57d281-566d7fb6c-jbk6q" event={"ID":"bc76fee9-e0bd-4ce6-9002-48182c70353c","Type":"ContainerDied","Data":"329fa3fdaeadc4ee62df3aa9ae90236f700a63d93b074bfc70b2f01af008f99b"} Nov 01 20:23:22.537098 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:22.537026 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-57d281-566d7fb6c-jbk6q" event={"ID":"bc76fee9-e0bd-4ce6-9002-48182c70353c","Type":"ContainerDied","Data":"ed38269da61e53fb36c06ffb5e1eedb226a4bec5297a077e15258e57020b70f2"} Nov 01 20:23:22.537098 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:22.537047 5935 scope.go:122] "RemoveContainer" containerID="329fa3fdaeadc4ee62df3aa9ae90236f700a63d93b074bfc70b2f01af008f99b" Nov 01 20:23:22.538561 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:22.538532 5935 generic.go:356] "Generic (PLEG): container finished" podID="a9e5d8fd-4547-409b-831a-ccfde839472c" containerID="646f333b57730c9e09def9e0a2bf98799cd3f78928d13c53683fa2456fc25346" exitCode=0 Nov 01 20:23:22.538684 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:22.538563 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-57d281-55d489b495-lvrd9" event={"ID":"a9e5d8fd-4547-409b-831a-ccfde839472c","Type":"ContainerDied","Data":"646f333b57730c9e09def9e0a2bf98799cd3f78928d13c53683fa2456fc25346"} Nov 01 20:23:22.538684 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:22.538599 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-57d281-55d489b495-lvrd9" Nov 01 20:23:22.538684 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:22.538605 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-57d281-55d489b495-lvrd9" event={"ID":"a9e5d8fd-4547-409b-831a-ccfde839472c","Type":"ContainerDied","Data":"f1c84e9352fd7d5c33243e55cf8a66bef8e257ca3afcc84dda87ffb373a0c2e2"} Nov 01 20:23:22.547652 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:22.547631 5935 scope.go:122] "RemoveContainer" containerID="329fa3fdaeadc4ee62df3aa9ae90236f700a63d93b074bfc70b2f01af008f99b" Nov 01 20:23:22.547957 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:23:22.547929 5935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"329fa3fdaeadc4ee62df3aa9ae90236f700a63d93b074bfc70b2f01af008f99b\": container with ID starting with 329fa3fdaeadc4ee62df3aa9ae90236f700a63d93b074bfc70b2f01af008f99b not found: ID does not exist" containerID="329fa3fdaeadc4ee62df3aa9ae90236f700a63d93b074bfc70b2f01af008f99b" Nov 01 20:23:22.548035 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:22.547969 5935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"329fa3fdaeadc4ee62df3aa9ae90236f700a63d93b074bfc70b2f01af008f99b"} err="failed to get container status \"329fa3fdaeadc4ee62df3aa9ae90236f700a63d93b074bfc70b2f01af008f99b\": rpc error: code = NotFound desc = could not find container \"329fa3fdaeadc4ee62df3aa9ae90236f700a63d93b074bfc70b2f01af008f99b\": container with ID starting with 329fa3fdaeadc4ee62df3aa9ae90236f700a63d93b074bfc70b2f01af008f99b not found: ID does not exist" Nov 01 20:23:22.548035 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:22.547992 5935 scope.go:122] "RemoveContainer" containerID="646f333b57730c9e09def9e0a2bf98799cd3f78928d13c53683fa2456fc25346" Nov 01 20:23:22.551538 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:22.551521 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-2babaef6/e2e-lifecycle-model-2babaef6-kserve-c7444bf-twlvv" Nov 01 20:23:22.557423 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:22.557401 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-57d281-55d489b495-lvrd9"] Nov 01 20:23:22.558771 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:22.558754 5935 scope.go:122] "RemoveContainer" containerID="646f333b57730c9e09def9e0a2bf98799cd3f78928d13c53683fa2456fc25346" Nov 01 20:23:22.559090 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:23:22.559069 5935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"646f333b57730c9e09def9e0a2bf98799cd3f78928d13c53683fa2456fc25346\": container with ID starting with 646f333b57730c9e09def9e0a2bf98799cd3f78928d13c53683fa2456fc25346 not found: ID does not exist" containerID="646f333b57730c9e09def9e0a2bf98799cd3f78928d13c53683fa2456fc25346" Nov 01 20:23:22.559166 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:22.559097 5935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"646f333b57730c9e09def9e0a2bf98799cd3f78928d13c53683fa2456fc25346"} err="failed to get container status \"646f333b57730c9e09def9e0a2bf98799cd3f78928d13c53683fa2456fc25346\": rpc error: code = NotFound desc = could not find container \"646f333b57730c9e09def9e0a2bf98799cd3f78928d13c53683fa2456fc25346\": container with ID starting with 646f333b57730c9e09def9e0a2bf98799cd3f78928d13c53683fa2456fc25346 not found: ID does not exist" Nov 01 20:23:22.560427 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:22.560410 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-57d281-55d489b495-lvrd9"] Nov 01 20:23:22.570021 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:22.570000 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-57d281-566d7fb6c-jbk6q"] Nov 01 20:23:22.572720 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:22.572700 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-57d281-566d7fb6c-jbk6q"] Nov 01 20:23:24.355945 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:24.355891 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9e5d8fd-4547-409b-831a-ccfde839472c" path="/var/lib/kubelet/pods/a9e5d8fd-4547-409b-831a-ccfde839472c/volumes" Nov 01 20:23:24.356666 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:24.356642 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc76fee9-e0bd-4ce6-9002-48182c70353c" path="/var/lib/kubelet/pods/bc76fee9-e0bd-4ce6-9002-48182c70353c/volumes" Nov 01 20:23:27.757636 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:27.757559 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-a-1908b3f5-kserve-5bbcdfcf99-d2nns"] Nov 01 20:23:27.758184 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:27.758169 5935 cpu_manager.go:414] "containerMap: removing container" podUID="869bb0c3-ff27-456b-a778-b7c4a4e85be2" containerName="payload-processing" Nov 01 20:23:27.758226 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:27.758185 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="869bb0c3-ff27-456b-a778-b7c4a4e85be2" containerName="payload-processing" Nov 01 20:23:27.758226 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:27.758204 5935 cpu_manager.go:414] "containerMap: removing container" podUID="bc76fee9-e0bd-4ce6-9002-48182c70353c" containerName="payload-pre-processing" Nov 01 20:23:27.758226 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:27.758209 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc76fee9-e0bd-4ce6-9002-48182c70353c" containerName="payload-pre-processing" Nov 01 20:23:27.758363 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:27.758230 5935 cpu_manager.go:414] "containerMap: removing container" podUID="a9e5d8fd-4547-409b-831a-ccfde839472c" containerName="payload-processing" Nov 01 20:23:27.758363 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:27.758236 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9e5d8fd-4547-409b-831a-ccfde839472c" containerName="payload-processing" Nov 01 20:23:27.758465 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:27.758449 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="869bb0c3-ff27-456b-a778-b7c4a4e85be2" containerName="payload-processing" Nov 01 20:23:27.758501 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:27.758473 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9e5d8fd-4547-409b-831a-ccfde839472c" containerName="payload-processing" Nov 01 20:23:27.758501 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:27.758483 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc76fee9-e0bd-4ce6-9002-48182c70353c" containerName="payload-pre-processing" Nov 01 20:23:27.763700 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:27.763679 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-a-1908b3f5-kserve-5bbcdfcf99-d2nns" Nov 01 20:23:27.775799 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:27.772749 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e27ae7\"/\"e2e-conflict-a-1908b3f5-kserve-self-signed-certs\"" Nov 01 20:23:27.775799 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:27.774767 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-a-1908b3f5-kserve-5bbcdfcf99-d2nns"] Nov 01 20:23:27.866787 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:27.866751 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cf4dde1-ffc3-4640-8b7f-65f3c4173e47-tls-certs\") pod \"e2e-conflict-a-1908b3f5-kserve-5bbcdfcf99-d2nns\" (UID: \"3cf4dde1-ffc3-4640-8b7f-65f3c4173e47\") " pod="e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-a-1908b3f5-kserve-5bbcdfcf99-d2nns" Nov 01 20:23:27.866976 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:27.866909 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cf4dde1-ffc3-4640-8b7f-65f3c4173e47-dshm\") pod \"e2e-conflict-a-1908b3f5-kserve-5bbcdfcf99-d2nns\" (UID: \"3cf4dde1-ffc3-4640-8b7f-65f3c4173e47\") " pod="e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-a-1908b3f5-kserve-5bbcdfcf99-d2nns" Nov 01 20:23:27.866976 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:27.866967 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cf4dde1-ffc3-4640-8b7f-65f3c4173e47-model-cache\") pod \"e2e-conflict-a-1908b3f5-kserve-5bbcdfcf99-d2nns\" (UID: \"3cf4dde1-ffc3-4640-8b7f-65f3c4173e47\") " pod="e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-a-1908b3f5-kserve-5bbcdfcf99-d2nns" Nov 01 20:23:27.867075 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:27.866996 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkqsz\" (UniqueName: \"kubernetes.io/projected/3cf4dde1-ffc3-4640-8b7f-65f3c4173e47-kube-api-access-pkqsz\") pod \"e2e-conflict-a-1908b3f5-kserve-5bbcdfcf99-d2nns\" (UID: \"3cf4dde1-ffc3-4640-8b7f-65f3c4173e47\") " pod="e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-a-1908b3f5-kserve-5bbcdfcf99-d2nns" Nov 01 20:23:27.867146 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:27.867121 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cf4dde1-ffc3-4640-8b7f-65f3c4173e47-home\") pod \"e2e-conflict-a-1908b3f5-kserve-5bbcdfcf99-d2nns\" (UID: \"3cf4dde1-ffc3-4640-8b7f-65f3c4173e47\") " pod="e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-a-1908b3f5-kserve-5bbcdfcf99-d2nns" Nov 01 20:23:27.867246 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:27.867228 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cf4dde1-ffc3-4640-8b7f-65f3c4173e47-tmp-dir\") pod \"e2e-conflict-a-1908b3f5-kserve-5bbcdfcf99-d2nns\" (UID: \"3cf4dde1-ffc3-4640-8b7f-65f3c4173e47\") " pod="e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-a-1908b3f5-kserve-5bbcdfcf99-d2nns" Nov 01 20:23:27.969817 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:27.968809 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cf4dde1-ffc3-4640-8b7f-65f3c4173e47-tls-certs\") pod \"e2e-conflict-a-1908b3f5-kserve-5bbcdfcf99-d2nns\" (UID: \"3cf4dde1-ffc3-4640-8b7f-65f3c4173e47\") " pod="e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-a-1908b3f5-kserve-5bbcdfcf99-d2nns" Nov 01 20:23:27.969817 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:27.968904 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cf4dde1-ffc3-4640-8b7f-65f3c4173e47-dshm\") pod \"e2e-conflict-a-1908b3f5-kserve-5bbcdfcf99-d2nns\" (UID: \"3cf4dde1-ffc3-4640-8b7f-65f3c4173e47\") " pod="e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-a-1908b3f5-kserve-5bbcdfcf99-d2nns" Nov 01 20:23:27.969817 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:27.968927 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cf4dde1-ffc3-4640-8b7f-65f3c4173e47-model-cache\") pod \"e2e-conflict-a-1908b3f5-kserve-5bbcdfcf99-d2nns\" (UID: \"3cf4dde1-ffc3-4640-8b7f-65f3c4173e47\") " pod="e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-a-1908b3f5-kserve-5bbcdfcf99-d2nns" Nov 01 20:23:27.969817 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:27.968951 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkqsz\" (UniqueName: \"kubernetes.io/projected/3cf4dde1-ffc3-4640-8b7f-65f3c4173e47-kube-api-access-pkqsz\") pod \"e2e-conflict-a-1908b3f5-kserve-5bbcdfcf99-d2nns\" (UID: \"3cf4dde1-ffc3-4640-8b7f-65f3c4173e47\") " pod="e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-a-1908b3f5-kserve-5bbcdfcf99-d2nns" Nov 01 20:23:27.969817 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:27.968995 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cf4dde1-ffc3-4640-8b7f-65f3c4173e47-home\") pod \"e2e-conflict-a-1908b3f5-kserve-5bbcdfcf99-d2nns\" (UID: \"3cf4dde1-ffc3-4640-8b7f-65f3c4173e47\") " pod="e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-a-1908b3f5-kserve-5bbcdfcf99-d2nns" Nov 01 20:23:27.969817 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:27.969040 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cf4dde1-ffc3-4640-8b7f-65f3c4173e47-tmp-dir\") pod \"e2e-conflict-a-1908b3f5-kserve-5bbcdfcf99-d2nns\" (UID: \"3cf4dde1-ffc3-4640-8b7f-65f3c4173e47\") " pod="e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-a-1908b3f5-kserve-5bbcdfcf99-d2nns" Nov 01 20:23:27.969817 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:27.969449 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cf4dde1-ffc3-4640-8b7f-65f3c4173e47-tmp-dir\") pod \"e2e-conflict-a-1908b3f5-kserve-5bbcdfcf99-d2nns\" (UID: \"3cf4dde1-ffc3-4640-8b7f-65f3c4173e47\") " pod="e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-a-1908b3f5-kserve-5bbcdfcf99-d2nns" Nov 01 20:23:27.970351 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:27.970033 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cf4dde1-ffc3-4640-8b7f-65f3c4173e47-model-cache\") pod \"e2e-conflict-a-1908b3f5-kserve-5bbcdfcf99-d2nns\" (UID: \"3cf4dde1-ffc3-4640-8b7f-65f3c4173e47\") " pod="e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-a-1908b3f5-kserve-5bbcdfcf99-d2nns" Nov 01 20:23:27.970351 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:27.970315 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cf4dde1-ffc3-4640-8b7f-65f3c4173e47-home\") pod \"e2e-conflict-a-1908b3f5-kserve-5bbcdfcf99-d2nns\" (UID: \"3cf4dde1-ffc3-4640-8b7f-65f3c4173e47\") " pod="e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-a-1908b3f5-kserve-5bbcdfcf99-d2nns" Nov 01 20:23:27.979233 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:27.979210 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkqsz\" (UniqueName: \"kubernetes.io/projected/3cf4dde1-ffc3-4640-8b7f-65f3c4173e47-kube-api-access-pkqsz\") pod \"e2e-conflict-a-1908b3f5-kserve-5bbcdfcf99-d2nns\" (UID: \"3cf4dde1-ffc3-4640-8b7f-65f3c4173e47\") " pod="e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-a-1908b3f5-kserve-5bbcdfcf99-d2nns" Nov 01 20:23:27.981081 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:27.981035 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cf4dde1-ffc3-4640-8b7f-65f3c4173e47-tls-certs\") pod \"e2e-conflict-a-1908b3f5-kserve-5bbcdfcf99-d2nns\" (UID: \"3cf4dde1-ffc3-4640-8b7f-65f3c4173e47\") " pod="e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-a-1908b3f5-kserve-5bbcdfcf99-d2nns" Nov 01 20:23:27.989107 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:27.984652 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cf4dde1-ffc3-4640-8b7f-65f3c4173e47-dshm\") pod \"e2e-conflict-a-1908b3f5-kserve-5bbcdfcf99-d2nns\" (UID: \"3cf4dde1-ffc3-4640-8b7f-65f3c4173e47\") " pod="e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-a-1908b3f5-kserve-5bbcdfcf99-d2nns" Nov 01 20:23:28.083209 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:28.083097 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-a-1908b3f5-kserve-5bbcdfcf99-d2nns" Nov 01 20:23:28.370988 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:28.370961 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-a-1908b3f5-kserve-5bbcdfcf99-d2nns"] Nov 01 20:23:28.371927 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:23:28.371906 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3cf4dde1_ffc3_4640_8b7f_65f3c4173e47.slice/crio-a94b645024c41cbaad61e36e166c10fc5a40d8d1beac3262ef88082834a49bca WatchSource:0}: Error finding container a94b645024c41cbaad61e36e166c10fc5a40d8d1beac3262ef88082834a49bca: Status 404 returned error can't find the container with id a94b645024c41cbaad61e36e166c10fc5a40d8d1beac3262ef88082834a49bca Nov 01 20:23:28.580553 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:28.580520 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-a-1908b3f5-kserve-5bbcdfcf99-d2nns" event={"ID":"3cf4dde1-ffc3-4640-8b7f-65f3c4173e47","Type":"ContainerStarted","Data":"87b6cc62e0bdf9019d9e003331a692d9e2946731c3beea283a93f6218eceb97f"} Nov 01 20:23:28.580718 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:28.580565 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-a-1908b3f5-kserve-5bbcdfcf99-d2nns" event={"ID":"3cf4dde1-ffc3-4640-8b7f-65f3c4173e47","Type":"ContainerStarted","Data":"a94b645024c41cbaad61e36e166c10fc5a40d8d1beac3262ef88082834a49bca"} Nov 01 20:23:28.598328 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:28.598269 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-a-1908b3f5-kserve-5bbcdfcf99-d2nns" podStartSLOduration=1.598256881 podStartE2EDuration="1.598256881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:23:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:23:28.595852144 +0000 UTC m=+1454.369012669" watchObservedRunningTime="2026-11-01 20:23:28.598256881 +0000 UTC m=+1454.371417392" Nov 01 20:23:29.084019 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:29.083986 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-a-1908b3f5-kserve-5bbcdfcf99-d2nns" Nov 01 20:23:35.845832 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:23:35.845785 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:23:38.083622 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:38.083586 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-a-1908b3f5-kserve-5bbcdfcf99-d2nns" Nov 01 20:23:38.096590 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:38.096564 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-a-1908b3f5-kserve-5bbcdfcf99-d2nns" Nov 01 20:23:38.698427 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:38.698398 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-a-1908b3f5-kserve-5bbcdfcf99-d2nns" Nov 01 20:23:41.676719 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:41.676686 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-82e760-openshift-default-5f697579fc-579fp"] Nov 01 20:23:41.680606 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:41.680585 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-82e760-openshift-default-5f697579fc-579fp" Nov 01 20:23:41.694209 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:41.694183 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-82e760-openshift-default-dockercfg-724hg\"" Nov 01 20:23:41.714611 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:41.714585 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-82e760-openshift-default-5f697579fc-579fp"] Nov 01 20:23:41.778254 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:41.778224 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b77275f-0120-4066-8e61-4d75130bb00a-istio-data\") pod \"e2e-noten-82e760-openshift-default-5f697579fc-579fp\" (UID: \"9b77275f-0120-4066-8e61-4d75130bb00a\") " pod="openshift-ingress/e2e-noten-82e760-openshift-default-5f697579fc-579fp" Nov 01 20:23:41.778572 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:41.778531 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b77275f-0120-4066-8e61-4d75130bb00a-istio-podinfo\") pod \"e2e-noten-82e760-openshift-default-5f697579fc-579fp\" (UID: \"9b77275f-0120-4066-8e61-4d75130bb00a\") " pod="openshift-ingress/e2e-noten-82e760-openshift-default-5f697579fc-579fp" Nov 01 20:23:41.778674 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:41.778662 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b77275f-0120-4066-8e61-4d75130bb00a-istio-envoy\") pod \"e2e-noten-82e760-openshift-default-5f697579fc-579fp\" (UID: \"9b77275f-0120-4066-8e61-4d75130bb00a\") " pod="openshift-ingress/e2e-noten-82e760-openshift-default-5f697579fc-579fp" Nov 01 20:23:41.778752 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:41.778737 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b77275f-0120-4066-8e61-4d75130bb00a-istiod-ca-cert\") pod \"e2e-noten-82e760-openshift-default-5f697579fc-579fp\" (UID: \"9b77275f-0120-4066-8e61-4d75130bb00a\") " pod="openshift-ingress/e2e-noten-82e760-openshift-default-5f697579fc-579fp" Nov 01 20:23:41.778858 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:41.778844 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b77275f-0120-4066-8e61-4d75130bb00a-workload-socket\") pod \"e2e-noten-82e760-openshift-default-5f697579fc-579fp\" (UID: \"9b77275f-0120-4066-8e61-4d75130bb00a\") " pod="openshift-ingress/e2e-noten-82e760-openshift-default-5f697579fc-579fp" Nov 01 20:23:41.778895 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:41.778887 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b77275f-0120-4066-8e61-4d75130bb00a-workload-certs\") pod \"e2e-noten-82e760-openshift-default-5f697579fc-579fp\" (UID: \"9b77275f-0120-4066-8e61-4d75130bb00a\") " pod="openshift-ingress/e2e-noten-82e760-openshift-default-5f697579fc-579fp" Nov 01 20:23:41.778960 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:41.778945 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b77275f-0120-4066-8e61-4d75130bb00a-istio-token\") pod \"e2e-noten-82e760-openshift-default-5f697579fc-579fp\" (UID: \"9b77275f-0120-4066-8e61-4d75130bb00a\") " pod="openshift-ingress/e2e-noten-82e760-openshift-default-5f697579fc-579fp" Nov 01 20:23:41.778995 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:41.778982 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn8tw\" (UniqueName: \"kubernetes.io/projected/9b77275f-0120-4066-8e61-4d75130bb00a-kube-api-access-zn8tw\") pod \"e2e-noten-82e760-openshift-default-5f697579fc-579fp\" (UID: \"9b77275f-0120-4066-8e61-4d75130bb00a\") " pod="openshift-ingress/e2e-noten-82e760-openshift-default-5f697579fc-579fp" Nov 01 20:23:41.779042 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:41.779028 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b77275f-0120-4066-8e61-4d75130bb00a-credential-socket\") pod \"e2e-noten-82e760-openshift-default-5f697579fc-579fp\" (UID: \"9b77275f-0120-4066-8e61-4d75130bb00a\") " pod="openshift-ingress/e2e-noten-82e760-openshift-default-5f697579fc-579fp" Nov 01 20:23:41.887141 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:41.887087 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b77275f-0120-4066-8e61-4d75130bb00a-istiod-ca-cert\") pod \"e2e-noten-82e760-openshift-default-5f697579fc-579fp\" (UID: \"9b77275f-0120-4066-8e61-4d75130bb00a\") " pod="openshift-ingress/e2e-noten-82e760-openshift-default-5f697579fc-579fp" Nov 01 20:23:41.887365 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:41.887212 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b77275f-0120-4066-8e61-4d75130bb00a-workload-socket\") pod \"e2e-noten-82e760-openshift-default-5f697579fc-579fp\" (UID: \"9b77275f-0120-4066-8e61-4d75130bb00a\") " pod="openshift-ingress/e2e-noten-82e760-openshift-default-5f697579fc-579fp" Nov 01 20:23:41.887365 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:41.887257 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b77275f-0120-4066-8e61-4d75130bb00a-workload-certs\") pod \"e2e-noten-82e760-openshift-default-5f697579fc-579fp\" (UID: \"9b77275f-0120-4066-8e61-4d75130bb00a\") " pod="openshift-ingress/e2e-noten-82e760-openshift-default-5f697579fc-579fp" Nov 01 20:23:41.887365 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:41.887334 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b77275f-0120-4066-8e61-4d75130bb00a-istio-token\") pod \"e2e-noten-82e760-openshift-default-5f697579fc-579fp\" (UID: \"9b77275f-0120-4066-8e61-4d75130bb00a\") " pod="openshift-ingress/e2e-noten-82e760-openshift-default-5f697579fc-579fp" Nov 01 20:23:41.887522 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:41.887372 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn8tw\" (UniqueName: \"kubernetes.io/projected/9b77275f-0120-4066-8e61-4d75130bb00a-kube-api-access-zn8tw\") pod \"e2e-noten-82e760-openshift-default-5f697579fc-579fp\" (UID: \"9b77275f-0120-4066-8e61-4d75130bb00a\") " pod="openshift-ingress/e2e-noten-82e760-openshift-default-5f697579fc-579fp" Nov 01 20:23:41.887522 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:41.887414 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b77275f-0120-4066-8e61-4d75130bb00a-credential-socket\") pod \"e2e-noten-82e760-openshift-default-5f697579fc-579fp\" (UID: \"9b77275f-0120-4066-8e61-4d75130bb00a\") " pod="openshift-ingress/e2e-noten-82e760-openshift-default-5f697579fc-579fp" Nov 01 20:23:41.887522 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:41.887476 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b77275f-0120-4066-8e61-4d75130bb00a-istio-data\") pod \"e2e-noten-82e760-openshift-default-5f697579fc-579fp\" (UID: \"9b77275f-0120-4066-8e61-4d75130bb00a\") " pod="openshift-ingress/e2e-noten-82e760-openshift-default-5f697579fc-579fp" Nov 01 20:23:41.887522 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:41.887501 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b77275f-0120-4066-8e61-4d75130bb00a-istio-podinfo\") pod \"e2e-noten-82e760-openshift-default-5f697579fc-579fp\" (UID: \"9b77275f-0120-4066-8e61-4d75130bb00a\") " pod="openshift-ingress/e2e-noten-82e760-openshift-default-5f697579fc-579fp" Nov 01 20:23:41.887688 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:41.887603 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b77275f-0120-4066-8e61-4d75130bb00a-istio-envoy\") pod \"e2e-noten-82e760-openshift-default-5f697579fc-579fp\" (UID: \"9b77275f-0120-4066-8e61-4d75130bb00a\") " pod="openshift-ingress/e2e-noten-82e760-openshift-default-5f697579fc-579fp" Nov 01 20:23:41.889162 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:41.889128 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b77275f-0120-4066-8e61-4d75130bb00a-istiod-ca-cert\") pod \"e2e-noten-82e760-openshift-default-5f697579fc-579fp\" (UID: \"9b77275f-0120-4066-8e61-4d75130bb00a\") " pod="openshift-ingress/e2e-noten-82e760-openshift-default-5f697579fc-579fp" Nov 01 20:23:41.890713 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:41.889415 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b77275f-0120-4066-8e61-4d75130bb00a-workload-socket\") pod \"e2e-noten-82e760-openshift-default-5f697579fc-579fp\" (UID: \"9b77275f-0120-4066-8e61-4d75130bb00a\") " pod="openshift-ingress/e2e-noten-82e760-openshift-default-5f697579fc-579fp" Nov 01 20:23:41.890713 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:41.889655 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b77275f-0120-4066-8e61-4d75130bb00a-workload-certs\") pod \"e2e-noten-82e760-openshift-default-5f697579fc-579fp\" (UID: \"9b77275f-0120-4066-8e61-4d75130bb00a\") " pod="openshift-ingress/e2e-noten-82e760-openshift-default-5f697579fc-579fp" Nov 01 20:23:41.890713 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:41.889896 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b77275f-0120-4066-8e61-4d75130bb00a-credential-socket\") pod \"e2e-noten-82e760-openshift-default-5f697579fc-579fp\" (UID: \"9b77275f-0120-4066-8e61-4d75130bb00a\") " pod="openshift-ingress/e2e-noten-82e760-openshift-default-5f697579fc-579fp" Nov 01 20:23:41.892786 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:41.892736 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b77275f-0120-4066-8e61-4d75130bb00a-istio-envoy\") pod \"e2e-noten-82e760-openshift-default-5f697579fc-579fp\" (UID: \"9b77275f-0120-4066-8e61-4d75130bb00a\") " pod="openshift-ingress/e2e-noten-82e760-openshift-default-5f697579fc-579fp" Nov 01 20:23:41.893923 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:41.893898 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b77275f-0120-4066-8e61-4d75130bb00a-istio-data\") pod \"e2e-noten-82e760-openshift-default-5f697579fc-579fp\" (UID: \"9b77275f-0120-4066-8e61-4d75130bb00a\") " pod="openshift-ingress/e2e-noten-82e760-openshift-default-5f697579fc-579fp" Nov 01 20:23:41.898346 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:41.898264 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b77275f-0120-4066-8e61-4d75130bb00a-istio-podinfo\") pod \"e2e-noten-82e760-openshift-default-5f697579fc-579fp\" (UID: \"9b77275f-0120-4066-8e61-4d75130bb00a\") " pod="openshift-ingress/e2e-noten-82e760-openshift-default-5f697579fc-579fp" Nov 01 20:23:41.901880 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:41.901853 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b77275f-0120-4066-8e61-4d75130bb00a-istio-token\") pod \"e2e-noten-82e760-openshift-default-5f697579fc-579fp\" (UID: \"9b77275f-0120-4066-8e61-4d75130bb00a\") " pod="openshift-ingress/e2e-noten-82e760-openshift-default-5f697579fc-579fp" Nov 01 20:23:41.902481 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:41.902459 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn8tw\" (UniqueName: \"kubernetes.io/projected/9b77275f-0120-4066-8e61-4d75130bb00a-kube-api-access-zn8tw\") pod \"e2e-noten-82e760-openshift-default-5f697579fc-579fp\" (UID: \"9b77275f-0120-4066-8e61-4d75130bb00a\") " pod="openshift-ingress/e2e-noten-82e760-openshift-default-5f697579fc-579fp" Nov 01 20:23:41.995811 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:41.995779 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-82e760-openshift-default-5f697579fc-579fp" Nov 01 20:23:42.291467 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:42.291365 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-82e760-openshift-default-5f697579fc-579fp"] Nov 01 20:23:42.294782 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:23:42.294746 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b77275f_0120_4066_8e61_4d75130bb00a.slice/crio-9d249df56a00a74d8597eb63e07d0217b76aa12ba28802bde3a45e8ae2b2146f WatchSource:0}: Error finding container 9d249df56a00a74d8597eb63e07d0217b76aa12ba28802bde3a45e8ae2b2146f: Status 404 returned error can't find the container with id 9d249df56a00a74d8597eb63e07d0217b76aa12ba28802bde3a45e8ae2b2146f Nov 01 20:23:42.298324 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:42.298250 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:23:42.298430 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:42.298350 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:23:42.298430 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:42.298404 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:23:42.722361 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:42.722325 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-82e760-openshift-default-5f697579fc-579fp" event={"ID":"9b77275f-0120-4066-8e61-4d75130bb00a","Type":"ContainerStarted","Data":"32abf72cc8759886e6060c6c7f44f8f21c12dfff11d47a08580acc7134500311"} Nov 01 20:23:42.722749 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:42.722390 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-82e760-openshift-default-5f697579fc-579fp" event={"ID":"9b77275f-0120-4066-8e61-4d75130bb00a","Type":"ContainerStarted","Data":"9d249df56a00a74d8597eb63e07d0217b76aa12ba28802bde3a45e8ae2b2146f"} Nov 01 20:23:42.745791 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:42.745743 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-82e760-openshift-default-5f697579fc-579fp" podStartSLOduration=1.7457306209999999 podStartE2EDuration="1.745730621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:23:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:23:42.740848915 +0000 UTC m=+1468.514009479" watchObservedRunningTime="2026-11-01 20:23:42.745730621 +0000 UTC m=+1468.518891132" Nov 01 20:23:42.996035 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:42.995864 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-82e760-openshift-default-5f697579fc-579fp" Nov 01 20:23:44.002221 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:44.002189 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-82e760-openshift-default-5f697579fc-579fp" Nov 01 20:23:44.746738 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:44.746709 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-82e760-openshift-default-5f697579fc-579fp" Nov 01 20:23:44.747551 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:23:44.747531 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-82e760-openshift-default-5f697579fc-579fp" Nov 01 20:24:00.418230 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:00.415235 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-82e760-openshift-default-5f697579fc-579fp"] Nov 01 20:24:00.418230 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:00.415855 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-82e760-openshift-default-5f697579fc-579fp" podUID="9b77275f-0120-4066-8e61-4d75130bb00a" containerName="istio-proxy" containerID="cri-o://32abf72cc8759886e6060c6c7f44f8f21c12dfff11d47a08580acc7134500311" gracePeriod=30 Nov 01 20:24:03.027893 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:03.027860 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-a-1908b3f5-kserve-5bbcdfcf99-d2nns"] Nov 01 20:24:05.693636 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:05.693612 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-82e760-openshift-default-5f697579fc-579fp" Nov 01 20:24:05.774637 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:05.774608 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b77275f-0120-4066-8e61-4d75130bb00a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b77275f-0120-4066-8e61-4d75130bb00a-istio-data\") pod \"9b77275f-0120-4066-8e61-4d75130bb00a\" (UID: \"9b77275f-0120-4066-8e61-4d75130bb00a\") " Nov 01 20:24:05.774823 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:05.774703 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b77275f-0120-4066-8e61-4d75130bb00a-istio-token\" (UniqueName: \"kubernetes.io/projected/9b77275f-0120-4066-8e61-4d75130bb00a-istio-token\") pod \"9b77275f-0120-4066-8e61-4d75130bb00a\" (UID: \"9b77275f-0120-4066-8e61-4d75130bb00a\") " Nov 01 20:24:05.774823 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:05.774761 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b77275f-0120-4066-8e61-4d75130bb00a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b77275f-0120-4066-8e61-4d75130bb00a-workload-certs\") pod \"9b77275f-0120-4066-8e61-4d75130bb00a\" (UID: \"9b77275f-0120-4066-8e61-4d75130bb00a\") " Nov 01 20:24:05.774823 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:05.774779 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b77275f-0120-4066-8e61-4d75130bb00a-kube-api-access-zn8tw\" (UniqueName: \"kubernetes.io/projected/9b77275f-0120-4066-8e61-4d75130bb00a-kube-api-access-zn8tw\") pod \"9b77275f-0120-4066-8e61-4d75130bb00a\" (UID: \"9b77275f-0120-4066-8e61-4d75130bb00a\") " Nov 01 20:24:05.774992 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:05.774846 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b77275f-0120-4066-8e61-4d75130bb00a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b77275f-0120-4066-8e61-4d75130bb00a-workload-socket\") pod \"9b77275f-0120-4066-8e61-4d75130bb00a\" (UID: \"9b77275f-0120-4066-8e61-4d75130bb00a\") " Nov 01 20:24:05.774992 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:05.774870 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b77275f-0120-4066-8e61-4d75130bb00a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b77275f-0120-4066-8e61-4d75130bb00a-istio-envoy\") pod \"9b77275f-0120-4066-8e61-4d75130bb00a\" (UID: \"9b77275f-0120-4066-8e61-4d75130bb00a\") " Nov 01 20:24:05.774992 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:05.774897 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9b77275f-0120-4066-8e61-4d75130bb00a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b77275f-0120-4066-8e61-4d75130bb00a-istio-podinfo\") pod \"9b77275f-0120-4066-8e61-4d75130bb00a\" (UID: \"9b77275f-0120-4066-8e61-4d75130bb00a\") " Nov 01 20:24:05.774992 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:05.774901 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b77275f-0120-4066-8e61-4d75130bb00a-istio-data" pod "9b77275f-0120-4066-8e61-4d75130bb00a" (UID: "9b77275f-0120-4066-8e61-4d75130bb00a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:24:05.774992 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:05.774924 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b77275f-0120-4066-8e61-4d75130bb00a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b77275f-0120-4066-8e61-4d75130bb00a-istiod-ca-cert\") pod \"9b77275f-0120-4066-8e61-4d75130bb00a\" (UID: \"9b77275f-0120-4066-8e61-4d75130bb00a\") " Nov 01 20:24:05.774992 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:05.774980 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b77275f-0120-4066-8e61-4d75130bb00a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b77275f-0120-4066-8e61-4d75130bb00a-credential-socket\") pod \"9b77275f-0120-4066-8e61-4d75130bb00a\" (UID: \"9b77275f-0120-4066-8e61-4d75130bb00a\") " Nov 01 20:24:05.775315 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:05.775038 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b77275f-0120-4066-8e61-4d75130bb00a-workload-certs" pod "9b77275f-0120-4066-8e61-4d75130bb00a" (UID: "9b77275f-0120-4066-8e61-4d75130bb00a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:24:05.775462 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:05.775382 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b77275f-0120-4066-8e61-4d75130bb00a-credential-socket" pod "9b77275f-0120-4066-8e61-4d75130bb00a" (UID: "9b77275f-0120-4066-8e61-4d75130bb00a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:24:05.775462 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:05.775426 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b77275f-0120-4066-8e61-4d75130bb00a-istiod-ca-cert" pod "9b77275f-0120-4066-8e61-4d75130bb00a" (UID: "9b77275f-0120-4066-8e61-4d75130bb00a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:24:05.775462 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:05.775438 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b77275f-0120-4066-8e61-4d75130bb00a-workload-socket" pod "9b77275f-0120-4066-8e61-4d75130bb00a" (UID: "9b77275f-0120-4066-8e61-4d75130bb00a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:24:05.775712 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:05.775544 5935 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b77275f-0120-4066-8e61-4d75130bb00a-istio-data\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:24:05.775712 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:05.775560 5935 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b77275f-0120-4066-8e61-4d75130bb00a-workload-certs\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:24:05.775712 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:05.775571 5935 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b77275f-0120-4066-8e61-4d75130bb00a-workload-socket\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:24:05.775712 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:05.775586 5935 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b77275f-0120-4066-8e61-4d75130bb00a-istiod-ca-cert\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:24:05.775712 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:05.775609 5935 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b77275f-0120-4066-8e61-4d75130bb00a-credential-socket\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:24:05.777193 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:05.777164 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b77275f-0120-4066-8e61-4d75130bb00a-istio-token" pod "9b77275f-0120-4066-8e61-4d75130bb00a" (UID: "9b77275f-0120-4066-8e61-4d75130bb00a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:24:05.777319 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:05.777188 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b77275f-0120-4066-8e61-4d75130bb00a-istio-envoy" pod "9b77275f-0120-4066-8e61-4d75130bb00a" (UID: "9b77275f-0120-4066-8e61-4d75130bb00a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:24:05.777319 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:05.777209 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b77275f-0120-4066-8e61-4d75130bb00a-kube-api-access-zn8tw" pod "9b77275f-0120-4066-8e61-4d75130bb00a" (UID: "9b77275f-0120-4066-8e61-4d75130bb00a"). InnerVolumeSpecName "kube-api-access-zn8tw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:24:05.777414 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:05.777313 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b77275f-0120-4066-8e61-4d75130bb00a-istio-podinfo" pod "9b77275f-0120-4066-8e61-4d75130bb00a" (UID: "9b77275f-0120-4066-8e61-4d75130bb00a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:24:05.851831 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:24:05.851788 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:24:05.877026 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:05.876993 5935 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b77275f-0120-4066-8e61-4d75130bb00a-istio-token\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:24:05.877198 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:05.877028 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zn8tw\" (UniqueName: \"kubernetes.io/projected/9b77275f-0120-4066-8e61-4d75130bb00a-kube-api-access-zn8tw\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:24:05.877198 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:05.877046 5935 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b77275f-0120-4066-8e61-4d75130bb00a-istio-envoy\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:24:05.877198 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:05.877062 5935 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b77275f-0120-4066-8e61-4d75130bb00a-istio-podinfo\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:24:05.919219 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:05.919185 5935 generic.go:356] "Generic (PLEG): container finished" podID="9b77275f-0120-4066-8e61-4d75130bb00a" containerID="32abf72cc8759886e6060c6c7f44f8f21c12dfff11d47a08580acc7134500311" exitCode=0 Nov 01 20:24:05.919411 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:05.919229 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-82e760-openshift-default-5f697579fc-579fp" Nov 01 20:24:05.919411 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:05.919248 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-82e760-openshift-default-5f697579fc-579fp" event={"ID":"9b77275f-0120-4066-8e61-4d75130bb00a","Type":"ContainerDied","Data":"32abf72cc8759886e6060c6c7f44f8f21c12dfff11d47a08580acc7134500311"} Nov 01 20:24:05.919411 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:05.919318 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-82e760-openshift-default-5f697579fc-579fp" event={"ID":"9b77275f-0120-4066-8e61-4d75130bb00a","Type":"ContainerDied","Data":"9d249df56a00a74d8597eb63e07d0217b76aa12ba28802bde3a45e8ae2b2146f"} Nov 01 20:24:05.919411 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:05.919340 5935 scope.go:122] "RemoveContainer" containerID="32abf72cc8759886e6060c6c7f44f8f21c12dfff11d47a08580acc7134500311" Nov 01 20:24:05.931454 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:05.931437 5935 scope.go:122] "RemoveContainer" containerID="32abf72cc8759886e6060c6c7f44f8f21c12dfff11d47a08580acc7134500311" Nov 01 20:24:05.931746 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:24:05.931726 5935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32abf72cc8759886e6060c6c7f44f8f21c12dfff11d47a08580acc7134500311\": container with ID starting with 32abf72cc8759886e6060c6c7f44f8f21c12dfff11d47a08580acc7134500311 not found: ID does not exist" containerID="32abf72cc8759886e6060c6c7f44f8f21c12dfff11d47a08580acc7134500311" Nov 01 20:24:05.931842 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:05.931753 5935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32abf72cc8759886e6060c6c7f44f8f21c12dfff11d47a08580acc7134500311"} err="failed to get container status \"32abf72cc8759886e6060c6c7f44f8f21c12dfff11d47a08580acc7134500311\": rpc error: code = NotFound desc = could not find container \"32abf72cc8759886e6060c6c7f44f8f21c12dfff11d47a08580acc7134500311\": container with ID starting with 32abf72cc8759886e6060c6c7f44f8f21c12dfff11d47a08580acc7134500311 not found: ID does not exist" Nov 01 20:24:05.943401 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:05.943376 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-82e760-openshift-default-5f697579fc-579fp"] Nov 01 20:24:05.949850 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:05.949795 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-82e760-openshift-default-5f697579fc-579fp"] Nov 01 20:24:06.354888 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:06.354813 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b77275f-0120-4066-8e61-4d75130bb00a" path="/var/lib/kubelet/pods/9b77275f-0120-4066-8e61-4d75130bb00a/volumes" Nov 01 20:24:12.731910 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:12.731861 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl"] Nov 01 20:24:12.733324 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:12.733279 5935 cpu_manager.go:414] "containerMap: removing container" podUID="9b77275f-0120-4066-8e61-4d75130bb00a" containerName="istio-proxy" Nov 01 20:24:12.733459 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:12.733446 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b77275f-0120-4066-8e61-4d75130bb00a" containerName="istio-proxy" Nov 01 20:24:12.733773 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:12.733749 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b77275f-0120-4066-8e61-4d75130bb00a" containerName="istio-proxy" Nov 01 20:24:12.739182 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:12.739158 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl" Nov 01 20:24:12.756913 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:12.756882 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl"] Nov 01 20:24:12.865103 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:12.865062 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16f53549-0a34-4825-a719-7ce0eff818ac-workload-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl\" (UID: \"16f53549-0a34-4825-a719-7ce0eff818ac\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl" Nov 01 20:24:12.865417 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:12.865399 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16f53549-0a34-4825-a719-7ce0eff818ac-credential-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl\" (UID: \"16f53549-0a34-4825-a719-7ce0eff818ac\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl" Nov 01 20:24:12.865571 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:12.865555 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16f53549-0a34-4825-a719-7ce0eff818ac-istio-envoy\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl\" (UID: \"16f53549-0a34-4825-a719-7ce0eff818ac\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl" Nov 01 20:24:12.865725 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:12.865710 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16f53549-0a34-4825-a719-7ce0eff818ac-istio-token\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl\" (UID: \"16f53549-0a34-4825-a719-7ce0eff818ac\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl" Nov 01 20:24:12.865875 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:12.865861 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzszd\" (UniqueName: \"kubernetes.io/projected/16f53549-0a34-4825-a719-7ce0eff818ac-kube-api-access-bzszd\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl\" (UID: \"16f53549-0a34-4825-a719-7ce0eff818ac\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl" Nov 01 20:24:12.865988 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:12.865973 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16f53549-0a34-4825-a719-7ce0eff818ac-istio-data\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl\" (UID: \"16f53549-0a34-4825-a719-7ce0eff818ac\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl" Nov 01 20:24:12.866085 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:12.866071 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16f53549-0a34-4825-a719-7ce0eff818ac-istiod-ca-cert\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl\" (UID: \"16f53549-0a34-4825-a719-7ce0eff818ac\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl" Nov 01 20:24:12.866266 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:12.866252 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16f53549-0a34-4825-a719-7ce0eff818ac-workload-certs\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl\" (UID: \"16f53549-0a34-4825-a719-7ce0eff818ac\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl" Nov 01 20:24:12.866417 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:12.866401 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16f53549-0a34-4825-a719-7ce0eff818ac-istio-podinfo\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl\" (UID: \"16f53549-0a34-4825-a719-7ce0eff818ac\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl" Nov 01 20:24:12.968211 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:12.968175 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16f53549-0a34-4825-a719-7ce0eff818ac-workload-certs\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl\" (UID: \"16f53549-0a34-4825-a719-7ce0eff818ac\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl" Nov 01 20:24:12.968417 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:12.968265 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16f53549-0a34-4825-a719-7ce0eff818ac-istio-podinfo\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl\" (UID: \"16f53549-0a34-4825-a719-7ce0eff818ac\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl" Nov 01 20:24:12.968417 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:12.968336 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16f53549-0a34-4825-a719-7ce0eff818ac-workload-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl\" (UID: \"16f53549-0a34-4825-a719-7ce0eff818ac\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl" Nov 01 20:24:12.968417 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:12.968394 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16f53549-0a34-4825-a719-7ce0eff818ac-credential-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl\" (UID: \"16f53549-0a34-4825-a719-7ce0eff818ac\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl" Nov 01 20:24:12.968600 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:12.968458 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16f53549-0a34-4825-a719-7ce0eff818ac-istio-envoy\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl\" (UID: \"16f53549-0a34-4825-a719-7ce0eff818ac\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl" Nov 01 20:24:12.968600 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:12.968514 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16f53549-0a34-4825-a719-7ce0eff818ac-istio-token\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl\" (UID: \"16f53549-0a34-4825-a719-7ce0eff818ac\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl" Nov 01 20:24:12.968600 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:12.968560 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzszd\" (UniqueName: \"kubernetes.io/projected/16f53549-0a34-4825-a719-7ce0eff818ac-kube-api-access-bzszd\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl\" (UID: \"16f53549-0a34-4825-a719-7ce0eff818ac\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl" Nov 01 20:24:12.968600 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:12.968581 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16f53549-0a34-4825-a719-7ce0eff818ac-istio-data\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl\" (UID: \"16f53549-0a34-4825-a719-7ce0eff818ac\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl" Nov 01 20:24:12.968600 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:12.968601 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16f53549-0a34-4825-a719-7ce0eff818ac-istiod-ca-cert\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl\" (UID: \"16f53549-0a34-4825-a719-7ce0eff818ac\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl" Nov 01 20:24:12.968863 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:12.968678 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16f53549-0a34-4825-a719-7ce0eff818ac-workload-certs\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl\" (UID: \"16f53549-0a34-4825-a719-7ce0eff818ac\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl" Nov 01 20:24:12.974464 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:12.969329 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16f53549-0a34-4825-a719-7ce0eff818ac-istiod-ca-cert\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl\" (UID: \"16f53549-0a34-4825-a719-7ce0eff818ac\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl" Nov 01 20:24:12.974464 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:12.971448 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16f53549-0a34-4825-a719-7ce0eff818ac-istio-envoy\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl\" (UID: \"16f53549-0a34-4825-a719-7ce0eff818ac\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl" Nov 01 20:24:12.974464 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:12.972421 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16f53549-0a34-4825-a719-7ce0eff818ac-istio-podinfo\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl\" (UID: \"16f53549-0a34-4825-a719-7ce0eff818ac\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl" Nov 01 20:24:12.974464 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:12.973609 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16f53549-0a34-4825-a719-7ce0eff818ac-istio-data\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl\" (UID: \"16f53549-0a34-4825-a719-7ce0eff818ac\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl" Nov 01 20:24:12.974464 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:12.973703 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16f53549-0a34-4825-a719-7ce0eff818ac-workload-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl\" (UID: \"16f53549-0a34-4825-a719-7ce0eff818ac\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl" Nov 01 20:24:12.974464 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:12.973714 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16f53549-0a34-4825-a719-7ce0eff818ac-credential-socket\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl\" (UID: \"16f53549-0a34-4825-a719-7ce0eff818ac\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl" Nov 01 20:24:12.977098 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:12.977074 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzszd\" (UniqueName: \"kubernetes.io/projected/16f53549-0a34-4825-a719-7ce0eff818ac-kube-api-access-bzszd\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl\" (UID: \"16f53549-0a34-4825-a719-7ce0eff818ac\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl" Nov 01 20:24:12.977336 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:12.977308 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16f53549-0a34-4825-a719-7ce0eff818ac-istio-token\") pod \"e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl\" (UID: \"16f53549-0a34-4825-a719-7ce0eff818ac\") " pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl" Nov 01 20:24:13.062558 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:13.062475 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl" Nov 01 20:24:13.360155 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:13.360124 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl"] Nov 01 20:24:13.361506 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:24:13.361478 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16f53549_0a34_4825_a719_7ce0eff818ac.slice/crio-ad713a49c3e0f501d1da7a631abbff41587353865710b2f576fab6298ca3b282 WatchSource:0}: Error finding container ad713a49c3e0f501d1da7a631abbff41587353865710b2f576fab6298ca3b282: Status 404 returned error can't find the container with id ad713a49c3e0f501d1da7a631abbff41587353865710b2f576fab6298ca3b282 Nov 01 20:24:13.364015 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:13.363983 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:24:13.364125 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:13.364058 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:24:13.364125 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:13.364101 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:24:13.983450 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:13.983411 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl" event={"ID":"16f53549-0a34-4825-a719-7ce0eff818ac","Type":"ContainerStarted","Data":"98a95c1f612cecb0a0ebe77340028fafeae5d7796fb3e419d9927fbc1491b62a"} Nov 01 20:24:13.983832 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:13.983479 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl" event={"ID":"16f53549-0a34-4825-a719-7ce0eff818ac","Type":"ContainerStarted","Data":"ad713a49c3e0f501d1da7a631abbff41587353865710b2f576fab6298ca3b282"} Nov 01 20:24:14.004916 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:14.004862 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl" podStartSLOduration=2.004848652 podStartE2EDuration="2.004848652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:24:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:24:14.002090204 +0000 UTC m=+1499.775250746" watchObservedRunningTime="2026-11-01 20:24:14.004848652 +0000 UTC m=+1499.778009167" Nov 01 20:24:14.063073 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:14.063045 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl" Nov 01 20:24:14.064701 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:14.064679 5935 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 01 20:24:14.064810 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:14.064729 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl" podUID="16f53549-0a34-4825-a719-7ce0eff818ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 01 20:24:14.727809 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:14.727773 5935 scope.go:122] "RemoveContainer" containerID="00f13d2bd84bfe82a8a74f5969cd780305ef7025ffe92e427ec3c2b6e32e2765" Nov 01 20:24:15.067744 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:15.067648 5935 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 01 20:24:15.067744 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:15.067713 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl" podUID="16f53549-0a34-4825-a719-7ce0eff818ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 01 20:24:16.067776 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:16.067741 5935 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 01 20:24:16.068407 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:16.068373 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl" podUID="16f53549-0a34-4825-a719-7ce0eff818ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 01 20:24:17.067657 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:17.067617 5935 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 01 20:24:17.067837 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:17.067677 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl" podUID="16f53549-0a34-4825-a719-7ce0eff818ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 01 20:24:18.059668 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:18.059608 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-a-1908b3f5-kserve-5bbcdfcf99-d2nns" podUID="3cf4dde1-ffc3-4640-8b7f-65f3c4173e47" containerName="main" containerID="cri-o://87b6cc62e0bdf9019d9e003331a692d9e2946731c3beea283a93f6218eceb97f" gracePeriod=45 Nov 01 20:24:18.065088 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:18.065057 5935 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 01 20:24:18.065325 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:18.065277 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl" podUID="16f53549-0a34-4825-a719-7ce0eff818ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 01 20:24:18.467313 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:18.466957 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-a-1908b3f5-kserve-5bbcdfcf99-d2nns" Nov 01 20:24:18.601152 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:18.601121 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cf4dde1-ffc3-4640-8b7f-65f3c4173e47-home\" (UniqueName: \"kubernetes.io/empty-dir/3cf4dde1-ffc3-4640-8b7f-65f3c4173e47-home\") pod \"3cf4dde1-ffc3-4640-8b7f-65f3c4173e47\" (UID: \"3cf4dde1-ffc3-4640-8b7f-65f3c4173e47\") " Nov 01 20:24:18.601364 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:18.601244 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cf4dde1-ffc3-4640-8b7f-65f3c4173e47-kube-api-access-pkqsz\" (UniqueName: \"kubernetes.io/projected/3cf4dde1-ffc3-4640-8b7f-65f3c4173e47-kube-api-access-pkqsz\") pod \"3cf4dde1-ffc3-4640-8b7f-65f3c4173e47\" (UID: \"3cf4dde1-ffc3-4640-8b7f-65f3c4173e47\") " Nov 01 20:24:18.601364 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:18.601302 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3cf4dde1-ffc3-4640-8b7f-65f3c4173e47-tls-certs\" (UniqueName: \"kubernetes.io/secret/3cf4dde1-ffc3-4640-8b7f-65f3c4173e47-tls-certs\") pod \"3cf4dde1-ffc3-4640-8b7f-65f3c4173e47\" (UID: \"3cf4dde1-ffc3-4640-8b7f-65f3c4173e47\") " Nov 01 20:24:18.601490 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:18.601368 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cf4dde1-ffc3-4640-8b7f-65f3c4173e47-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cf4dde1-ffc3-4640-8b7f-65f3c4173e47-model-cache\") pod \"3cf4dde1-ffc3-4640-8b7f-65f3c4173e47\" (UID: \"3cf4dde1-ffc3-4640-8b7f-65f3c4173e47\") " Nov 01 20:24:18.601490 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:18.601412 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cf4dde1-ffc3-4640-8b7f-65f3c4173e47-home" pod "3cf4dde1-ffc3-4640-8b7f-65f3c4173e47" (UID: "3cf4dde1-ffc3-4640-8b7f-65f3c4173e47"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:24:18.601490 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:18.601467 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cf4dde1-ffc3-4640-8b7f-65f3c4173e47-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cf4dde1-ffc3-4640-8b7f-65f3c4173e47-tmp-dir\") pod \"3cf4dde1-ffc3-4640-8b7f-65f3c4173e47\" (UID: \"3cf4dde1-ffc3-4640-8b7f-65f3c4173e47\") " Nov 01 20:24:18.601632 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:18.601523 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3cf4dde1-ffc3-4640-8b7f-65f3c4173e47-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cf4dde1-ffc3-4640-8b7f-65f3c4173e47-dshm\") pod \"3cf4dde1-ffc3-4640-8b7f-65f3c4173e47\" (UID: \"3cf4dde1-ffc3-4640-8b7f-65f3c4173e47\") " Nov 01 20:24:18.601691 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:18.601627 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cf4dde1-ffc3-4640-8b7f-65f3c4173e47-model-cache" pod "3cf4dde1-ffc3-4640-8b7f-65f3c4173e47" (UID: "3cf4dde1-ffc3-4640-8b7f-65f3c4173e47"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:24:18.601752 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:18.601708 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cf4dde1-ffc3-4640-8b7f-65f3c4173e47-tmp-dir" pod "3cf4dde1-ffc3-4640-8b7f-65f3c4173e47" (UID: "3cf4dde1-ffc3-4640-8b7f-65f3c4173e47"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:24:18.602010 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:18.601994 5935 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3cf4dde1-ffc3-4640-8b7f-65f3c4173e47-home\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:24:18.602069 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:18.602012 5935 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3cf4dde1-ffc3-4640-8b7f-65f3c4173e47-model-cache\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:24:18.602069 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:18.602025 5935 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3cf4dde1-ffc3-4640-8b7f-65f3c4173e47-tmp-dir\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:24:18.603498 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:18.603466 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cf4dde1-ffc3-4640-8b7f-65f3c4173e47-kube-api-access-pkqsz" pod "3cf4dde1-ffc3-4640-8b7f-65f3c4173e47" (UID: "3cf4dde1-ffc3-4640-8b7f-65f3c4173e47"). InnerVolumeSpecName "kube-api-access-pkqsz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:24:18.603755 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:18.603736 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3cf4dde1-ffc3-4640-8b7f-65f3c4173e47-dshm" pod "3cf4dde1-ffc3-4640-8b7f-65f3c4173e47" (UID: "3cf4dde1-ffc3-4640-8b7f-65f3c4173e47"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:24:18.603833 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:18.603752 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cf4dde1-ffc3-4640-8b7f-65f3c4173e47-tls-certs" pod "3cf4dde1-ffc3-4640-8b7f-65f3c4173e47" (UID: "3cf4dde1-ffc3-4640-8b7f-65f3c4173e47"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:24:18.703610 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:18.703578 5935 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3cf4dde1-ffc3-4640-8b7f-65f3c4173e47-dshm\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:24:18.703610 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:18.703609 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkqsz\" (UniqueName: \"kubernetes.io/projected/3cf4dde1-ffc3-4640-8b7f-65f3c4173e47-kube-api-access-pkqsz\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:24:18.703803 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:18.703620 5935 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cf4dde1-ffc3-4640-8b7f-65f3c4173e47-tls-certs\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:24:18.966056 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:18.964935 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x"] Nov 01 20:24:18.966056 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:18.965553 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x" podUID="3b63dca7-1aeb-40c5-8b7c-328578a58adf" containerName="istio-proxy" containerID="cri-o://caf0c3b20c44c875936c8eff8a6a59738c096cad97f7651a9bee823ebb5d9bb5" gracePeriod=30 Nov 01 20:24:19.018179 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:19.018139 5935 generic.go:356] "Generic (PLEG): container finished" podID="3cf4dde1-ffc3-4640-8b7f-65f3c4173e47" containerID="87b6cc62e0bdf9019d9e003331a692d9e2946731c3beea283a93f6218eceb97f" exitCode=0 Nov 01 20:24:19.018389 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:19.018211 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-a-1908b3f5-kserve-5bbcdfcf99-d2nns" Nov 01 20:24:19.018389 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:19.018216 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-a-1908b3f5-kserve-5bbcdfcf99-d2nns" event={"ID":"3cf4dde1-ffc3-4640-8b7f-65f3c4173e47","Type":"ContainerDied","Data":"87b6cc62e0bdf9019d9e003331a692d9e2946731c3beea283a93f6218eceb97f"} Nov 01 20:24:19.018389 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:19.018262 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-a-1908b3f5-kserve-5bbcdfcf99-d2nns" event={"ID":"3cf4dde1-ffc3-4640-8b7f-65f3c4173e47","Type":"ContainerDied","Data":"a94b645024c41cbaad61e36e166c10fc5a40d8d1beac3262ef88082834a49bca"} Nov 01 20:24:19.018389 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:19.018279 5935 scope.go:122] "RemoveContainer" containerID="87b6cc62e0bdf9019d9e003331a692d9e2946731c3beea283a93f6218eceb97f" Nov 01 20:24:19.029556 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:19.029524 5935 scope.go:122] "RemoveContainer" containerID="87b6cc62e0bdf9019d9e003331a692d9e2946731c3beea283a93f6218eceb97f" Nov 01 20:24:19.029867 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:24:19.029841 5935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87b6cc62e0bdf9019d9e003331a692d9e2946731c3beea283a93f6218eceb97f\": container with ID starting with 87b6cc62e0bdf9019d9e003331a692d9e2946731c3beea283a93f6218eceb97f not found: ID does not exist" containerID="87b6cc62e0bdf9019d9e003331a692d9e2946731c3beea283a93f6218eceb97f" Nov 01 20:24:19.029938 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:19.029867 5935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87b6cc62e0bdf9019d9e003331a692d9e2946731c3beea283a93f6218eceb97f"} err="failed to get container status \"87b6cc62e0bdf9019d9e003331a692d9e2946731c3beea283a93f6218eceb97f\": rpc error: code = NotFound desc = could not find container \"87b6cc62e0bdf9019d9e003331a692d9e2946731c3beea283a93f6218eceb97f\": container with ID starting with 87b6cc62e0bdf9019d9e003331a692d9e2946731c3beea283a93f6218eceb97f not found: ID does not exist" Nov 01 20:24:19.041269 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:19.041245 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-a-1908b3f5-kserve-5bbcdfcf99-d2nns"] Nov 01 20:24:19.055309 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:19.052663 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-e27ae7/e2e-conflict-a-1908b3f5-kserve-5bbcdfcf99-d2nns"] Nov 01 20:24:19.073359 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:19.071418 5935 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 01 20:24:19.073359 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:19.071484 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl" podUID="16f53549-0a34-4825-a719-7ce0eff818ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 01 20:24:20.063161 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:20.063124 5935 patch_prober.go:29] interesting pod/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Nov 01 20:24:20.063559 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:20.063178 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl" podUID="16f53549-0a34-4825-a719-7ce0eff818ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Nov 01 20:24:20.356560 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:20.356487 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cf4dde1-ffc3-4640-8b7f-65f3c4173e47" path="/var/lib/kubelet/pods/3cf4dde1-ffc3-4640-8b7f-65f3c4173e47/volumes" Nov 01 20:24:21.154758 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:21.154727 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl" Nov 01 20:24:21.155474 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:21.155412 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl" Nov 01 20:24:21.156141 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:21.156123 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl" Nov 01 20:24:24.273128 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:24.273101 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x" Nov 01 20:24:24.335152 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:24.335119 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b63dca7-1aeb-40c5-8b7c-328578a58adf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b63dca7-1aeb-40c5-8b7c-328578a58adf-istio-envoy\") pod \"3b63dca7-1aeb-40c5-8b7c-328578a58adf\" (UID: \"3b63dca7-1aeb-40c5-8b7c-328578a58adf\") " Nov 01 20:24:24.335328 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:24.335264 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b63dca7-1aeb-40c5-8b7c-328578a58adf-istio-token\" (UniqueName: \"kubernetes.io/projected/3b63dca7-1aeb-40c5-8b7c-328578a58adf-istio-token\") pod \"3b63dca7-1aeb-40c5-8b7c-328578a58adf\" (UID: \"3b63dca7-1aeb-40c5-8b7c-328578a58adf\") " Nov 01 20:24:24.335328 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:24.335306 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b63dca7-1aeb-40c5-8b7c-328578a58adf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b63dca7-1aeb-40c5-8b7c-328578a58adf-workload-certs\") pod \"3b63dca7-1aeb-40c5-8b7c-328578a58adf\" (UID: \"3b63dca7-1aeb-40c5-8b7c-328578a58adf\") " Nov 01 20:24:24.335469 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:24.335352 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b63dca7-1aeb-40c5-8b7c-328578a58adf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b63dca7-1aeb-40c5-8b7c-328578a58adf-istiod-ca-cert\") pod \"3b63dca7-1aeb-40c5-8b7c-328578a58adf\" (UID: \"3b63dca7-1aeb-40c5-8b7c-328578a58adf\") " Nov 01 20:24:24.335469 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:24.335385 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b63dca7-1aeb-40c5-8b7c-328578a58adf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b63dca7-1aeb-40c5-8b7c-328578a58adf-credential-socket\") pod \"3b63dca7-1aeb-40c5-8b7c-328578a58adf\" (UID: \"3b63dca7-1aeb-40c5-8b7c-328578a58adf\") " Nov 01 20:24:24.335469 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:24.335406 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b63dca7-1aeb-40c5-8b7c-328578a58adf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b63dca7-1aeb-40c5-8b7c-328578a58adf-istio-data\") pod \"3b63dca7-1aeb-40c5-8b7c-328578a58adf\" (UID: \"3b63dca7-1aeb-40c5-8b7c-328578a58adf\") " Nov 01 20:24:24.335469 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:24.335462 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b63dca7-1aeb-40c5-8b7c-328578a58adf-kube-api-access-46zxz\" (UniqueName: \"kubernetes.io/projected/3b63dca7-1aeb-40c5-8b7c-328578a58adf-kube-api-access-46zxz\") pod \"3b63dca7-1aeb-40c5-8b7c-328578a58adf\" (UID: \"3b63dca7-1aeb-40c5-8b7c-328578a58adf\") " Nov 01 20:24:24.335671 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:24.335486 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b63dca7-1aeb-40c5-8b7c-328578a58adf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b63dca7-1aeb-40c5-8b7c-328578a58adf-workload-socket\") pod \"3b63dca7-1aeb-40c5-8b7c-328578a58adf\" (UID: \"3b63dca7-1aeb-40c5-8b7c-328578a58adf\") " Nov 01 20:24:24.335671 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:24.335527 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3b63dca7-1aeb-40c5-8b7c-328578a58adf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b63dca7-1aeb-40c5-8b7c-328578a58adf-istio-podinfo\") pod \"3b63dca7-1aeb-40c5-8b7c-328578a58adf\" (UID: \"3b63dca7-1aeb-40c5-8b7c-328578a58adf\") " Nov 01 20:24:24.336161 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:24.336024 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b63dca7-1aeb-40c5-8b7c-328578a58adf-workload-certs" pod "3b63dca7-1aeb-40c5-8b7c-328578a58adf" (UID: "3b63dca7-1aeb-40c5-8b7c-328578a58adf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:24:24.336262 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:24.336161 5935 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b63dca7-1aeb-40c5-8b7c-328578a58adf-workload-certs\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:24:24.336368 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:24.336263 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b63dca7-1aeb-40c5-8b7c-328578a58adf-workload-socket" pod "3b63dca7-1aeb-40c5-8b7c-328578a58adf" (UID: "3b63dca7-1aeb-40c5-8b7c-328578a58adf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:24:24.336467 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:24.336433 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b63dca7-1aeb-40c5-8b7c-328578a58adf-istiod-ca-cert" pod "3b63dca7-1aeb-40c5-8b7c-328578a58adf" (UID: "3b63dca7-1aeb-40c5-8b7c-328578a58adf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:24:24.336811 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:24.336764 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b63dca7-1aeb-40c5-8b7c-328578a58adf-istio-data" pod "3b63dca7-1aeb-40c5-8b7c-328578a58adf" (UID: "3b63dca7-1aeb-40c5-8b7c-328578a58adf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:24:24.337705 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:24.337683 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b63dca7-1aeb-40c5-8b7c-328578a58adf-credential-socket" pod "3b63dca7-1aeb-40c5-8b7c-328578a58adf" (UID: "3b63dca7-1aeb-40c5-8b7c-328578a58adf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:24:24.338465 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:24.338439 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b63dca7-1aeb-40c5-8b7c-328578a58adf-istio-envoy" pod "3b63dca7-1aeb-40c5-8b7c-328578a58adf" (UID: "3b63dca7-1aeb-40c5-8b7c-328578a58adf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:24:24.338660 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:24.338637 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b63dca7-1aeb-40c5-8b7c-328578a58adf-kube-api-access-46zxz" pod "3b63dca7-1aeb-40c5-8b7c-328578a58adf" (UID: "3b63dca7-1aeb-40c5-8b7c-328578a58adf"). InnerVolumeSpecName "kube-api-access-46zxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:24:24.338863 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:24.338843 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b63dca7-1aeb-40c5-8b7c-328578a58adf-istio-podinfo" pod "3b63dca7-1aeb-40c5-8b7c-328578a58adf" (UID: "3b63dca7-1aeb-40c5-8b7c-328578a58adf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:24:24.338916 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:24.338846 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b63dca7-1aeb-40c5-8b7c-328578a58adf-istio-token" pod "3b63dca7-1aeb-40c5-8b7c-328578a58adf" (UID: "3b63dca7-1aeb-40c5-8b7c-328578a58adf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:24:24.438078 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:24.438031 5935 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b63dca7-1aeb-40c5-8b7c-328578a58adf-istio-token\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:24:24.438078 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:24.438066 5935 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b63dca7-1aeb-40c5-8b7c-328578a58adf-istiod-ca-cert\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:24:24.438344 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:24.438085 5935 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b63dca7-1aeb-40c5-8b7c-328578a58adf-credential-socket\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:24:24.438344 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:24.438104 5935 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b63dca7-1aeb-40c5-8b7c-328578a58adf-istio-data\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:24:24.438344 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:24.438120 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46zxz\" (UniqueName: \"kubernetes.io/projected/3b63dca7-1aeb-40c5-8b7c-328578a58adf-kube-api-access-46zxz\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:24:24.438344 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:24.438135 5935 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b63dca7-1aeb-40c5-8b7c-328578a58adf-workload-socket\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:24:24.438344 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:24.438149 5935 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b63dca7-1aeb-40c5-8b7c-328578a58adf-istio-podinfo\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:24:24.438344 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:24.438177 5935 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b63dca7-1aeb-40c5-8b7c-328578a58adf-istio-envoy\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:24:25.078811 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:25.078771 5935 generic.go:356] "Generic (PLEG): container finished" podID="3b63dca7-1aeb-40c5-8b7c-328578a58adf" containerID="caf0c3b20c44c875936c8eff8a6a59738c096cad97f7651a9bee823ebb5d9bb5" exitCode=0 Nov 01 20:24:25.078811 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:25.078802 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x" event={"ID":"3b63dca7-1aeb-40c5-8b7c-328578a58adf","Type":"ContainerDied","Data":"caf0c3b20c44c875936c8eff8a6a59738c096cad97f7651a9bee823ebb5d9bb5"} Nov 01 20:24:25.079063 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:25.078822 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x" Nov 01 20:24:25.079063 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:25.078857 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x" event={"ID":"3b63dca7-1aeb-40c5-8b7c-328578a58adf","Type":"ContainerDied","Data":"209b305956d735e360d90ba27ee6dee124cbb7ae99cedf4b3ea1e6312da29e36"} Nov 01 20:24:25.079063 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:25.078884 5935 scope.go:122] "RemoveContainer" containerID="caf0c3b20c44c875936c8eff8a6a59738c096cad97f7651a9bee823ebb5d9bb5" Nov 01 20:24:25.090532 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:25.090518 5935 scope.go:122] "RemoveContainer" containerID="caf0c3b20c44c875936c8eff8a6a59738c096cad97f7651a9bee823ebb5d9bb5" Nov 01 20:24:25.090741 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:24:25.090722 5935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"caf0c3b20c44c875936c8eff8a6a59738c096cad97f7651a9bee823ebb5d9bb5\": container with ID starting with caf0c3b20c44c875936c8eff8a6a59738c096cad97f7651a9bee823ebb5d9bb5 not found: ID does not exist" containerID="caf0c3b20c44c875936c8eff8a6a59738c096cad97f7651a9bee823ebb5d9bb5" Nov 01 20:24:25.090792 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:25.090749 5935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"caf0c3b20c44c875936c8eff8a6a59738c096cad97f7651a9bee823ebb5d9bb5"} err="failed to get container status \"caf0c3b20c44c875936c8eff8a6a59738c096cad97f7651a9bee823ebb5d9bb5\": rpc error: code = NotFound desc = could not find container \"caf0c3b20c44c875936c8eff8a6a59738c096cad97f7651a9bee823ebb5d9bb5\": container with ID starting with caf0c3b20c44c875936c8eff8a6a59738c096cad97f7651a9bee823ebb5d9bb5 not found: ID does not exist" Nov 01 20:24:25.098886 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:25.098596 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x"] Nov 01 20:24:25.101621 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:25.101590 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-29b57c-openshift-default-594b494869-h8v6x"] Nov 01 20:24:26.354703 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:26.354664 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b63dca7-1aeb-40c5-8b7c-328578a58adf" path="/var/lib/kubelet/pods/3b63dca7-1aeb-40c5-8b7c-328578a58adf/volumes" Nov 01 20:24:35.846268 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:24:35.846214 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:24:42.126494 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:42.126458 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e27ae7/distinct-2-w2-e27ae7-kserve-6bf4d46475-jdjhp"] Nov 01 20:24:42.134469 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:42.134441 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e27ae7/distinct-w2-e27ae7-kserve-7df8547d56-9qgdk"] Nov 01 20:24:48.328275 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:48.328241 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-e27ae7-7c877fb66c-hglrt"] Nov 01 20:24:53.329433 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:53.329374 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e27ae7-7c877fb66c-hglrt" podUID="b7d68b3d-ba87-4c71-9399-989f83e198b4" containerName="payload-pre-processing" containerID="cri-o://a8d6eee555e6d0a20c087b0b50848246eb6b8dcb0b4dc2f752041511e25246fc" gracePeriod=25 Nov 01 20:24:56.703029 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:56.702945 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e27ae7-7c877fb66c-hglrt" podUID="b7d68b3d-ba87-4c71-9399-989f83e198b4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 20:24:57.149223 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:57.148957 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-e27ae7/distinct-2-w2-e27ae7-kserve-6bf4d46475-jdjhp" podUID="b916d612-a355-4525-8143-9e3a47ead6d0" containerName="main" containerID="cri-o://0ca1f24ac8ccb1cf5520880d24e4719caf99cc6c1de37d73d8f2239df2d2190d" gracePeriod=45 Nov 01 20:24:57.161050 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:57.160988 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-e27ae7/distinct-w2-e27ae7-kserve-7df8547d56-9qgdk" podUID="4a948974-537a-49da-85e7-60adf68d1655" containerName="main" containerID="cri-o://cae5df9029be86bfb8b61d6230defafa4ae76e893262625fd835f376329609f6" gracePeriod=45 Nov 01 20:24:57.167693 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:57.167647 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e27ae7/distinct-w2-e27ae7-kserve-7df8547d56-9qgdk" podUID="4a948974-537a-49da-85e7-60adf68d1655" containerName="main" probeResult="failure" output="Get \"https://10.133.0.67:8000/ready\": EOF" Nov 01 20:24:57.273479 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:57.273447 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e27ae7/distinct-2-w2-e27ae7-kserve-6bf4d46475-jdjhp" podUID="b916d612-a355-4525-8143-9e3a47ead6d0" containerName="main" probeResult="failure" output="Get \"https://10.133.0.68:8000/ready\": dial tcp 10.133.0.68:8000: connect: connection refused" Nov 01 20:24:57.293232 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:57.293198 5935 generic.go:356] "Generic (PLEG): container finished" podID="4a948974-537a-49da-85e7-60adf68d1655" containerID="cae5df9029be86bfb8b61d6230defafa4ae76e893262625fd835f376329609f6" exitCode=0 Nov 01 20:24:57.293419 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:57.293264 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e27ae7/distinct-w2-e27ae7-kserve-7df8547d56-9qgdk" event={"ID":"4a948974-537a-49da-85e7-60adf68d1655","Type":"ContainerDied","Data":"cae5df9029be86bfb8b61d6230defafa4ae76e893262625fd835f376329609f6"} Nov 01 20:24:57.295414 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:57.295387 5935 generic.go:356] "Generic (PLEG): container finished" podID="b916d612-a355-4525-8143-9e3a47ead6d0" containerID="0ca1f24ac8ccb1cf5520880d24e4719caf99cc6c1de37d73d8f2239df2d2190d" exitCode=0 Nov 01 20:24:57.295540 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:57.295458 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e27ae7/distinct-2-w2-e27ae7-kserve-6bf4d46475-jdjhp" event={"ID":"b916d612-a355-4525-8143-9e3a47ead6d0","Type":"ContainerDied","Data":"0ca1f24ac8ccb1cf5520880d24e4719caf99cc6c1de37d73d8f2239df2d2190d"} Nov 01 20:24:57.457226 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:57.457201 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e27ae7/distinct-2-w2-e27ae7-kserve-6bf4d46475-jdjhp" Nov 01 20:24:57.460932 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:57.460901 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e27ae7/distinct-w2-e27ae7-kserve-7df8547d56-9qgdk" Nov 01 20:24:57.598062 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:57.598027 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b916d612-a355-4525-8143-9e3a47ead6d0-tls-certs\" (UniqueName: \"kubernetes.io/secret/b916d612-a355-4525-8143-9e3a47ead6d0-tls-certs\") pod \"b916d612-a355-4525-8143-9e3a47ead6d0\" (UID: \"b916d612-a355-4525-8143-9e3a47ead6d0\") " Nov 01 20:24:57.598225 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:57.598126 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a948974-537a-49da-85e7-60adf68d1655-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a948974-537a-49da-85e7-60adf68d1655-model-cache\") pod \"4a948974-537a-49da-85e7-60adf68d1655\" (UID: \"4a948974-537a-49da-85e7-60adf68d1655\") " Nov 01 20:24:57.598225 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:57.598217 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b916d612-a355-4525-8143-9e3a47ead6d0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b916d612-a355-4525-8143-9e3a47ead6d0-dshm\") pod \"b916d612-a355-4525-8143-9e3a47ead6d0\" (UID: \"b916d612-a355-4525-8143-9e3a47ead6d0\") " Nov 01 20:24:57.598356 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:57.598260 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b916d612-a355-4525-8143-9e3a47ead6d0-home\" (UniqueName: \"kubernetes.io/empty-dir/b916d612-a355-4525-8143-9e3a47ead6d0-home\") pod \"b916d612-a355-4525-8143-9e3a47ead6d0\" (UID: \"b916d612-a355-4525-8143-9e3a47ead6d0\") " Nov 01 20:24:57.598356 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:57.598338 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a948974-537a-49da-85e7-60adf68d1655-kube-api-access-dqchf\" (UniqueName: \"kubernetes.io/projected/4a948974-537a-49da-85e7-60adf68d1655-kube-api-access-dqchf\") pod \"4a948974-537a-49da-85e7-60adf68d1655\" (UID: \"4a948974-537a-49da-85e7-60adf68d1655\") " Nov 01 20:24:57.598467 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:57.598369 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b916d612-a355-4525-8143-9e3a47ead6d0-kube-api-access-kv6lk\" (UniqueName: \"kubernetes.io/projected/b916d612-a355-4525-8143-9e3a47ead6d0-kube-api-access-kv6lk\") pod \"b916d612-a355-4525-8143-9e3a47ead6d0\" (UID: \"b916d612-a355-4525-8143-9e3a47ead6d0\") " Nov 01 20:24:57.598467 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:57.598423 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a948974-537a-49da-85e7-60adf68d1655-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a948974-537a-49da-85e7-60adf68d1655-tmp-dir\") pod \"4a948974-537a-49da-85e7-60adf68d1655\" (UID: \"4a948974-537a-49da-85e7-60adf68d1655\") " Nov 01 20:24:57.598467 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:57.598429 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a948974-537a-49da-85e7-60adf68d1655-model-cache" pod "4a948974-537a-49da-85e7-60adf68d1655" (UID: "4a948974-537a-49da-85e7-60adf68d1655"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:24:57.598467 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:57.598453 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a948974-537a-49da-85e7-60adf68d1655-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a948974-537a-49da-85e7-60adf68d1655-dshm\") pod \"4a948974-537a-49da-85e7-60adf68d1655\" (UID: \"4a948974-537a-49da-85e7-60adf68d1655\") " Nov 01 20:24:57.598686 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:57.598531 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a948974-537a-49da-85e7-60adf68d1655-tls-certs\" (UniqueName: \"kubernetes.io/secret/4a948974-537a-49da-85e7-60adf68d1655-tls-certs\") pod \"4a948974-537a-49da-85e7-60adf68d1655\" (UID: \"4a948974-537a-49da-85e7-60adf68d1655\") " Nov 01 20:24:57.598686 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:57.598556 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b916d612-a355-4525-8143-9e3a47ead6d0-home" pod "b916d612-a355-4525-8143-9e3a47ead6d0" (UID: "b916d612-a355-4525-8143-9e3a47ead6d0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:24:57.598686 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:57.598578 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b916d612-a355-4525-8143-9e3a47ead6d0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b916d612-a355-4525-8143-9e3a47ead6d0-tmp-dir\") pod \"b916d612-a355-4525-8143-9e3a47ead6d0\" (UID: \"b916d612-a355-4525-8143-9e3a47ead6d0\") " Nov 01 20:24:57.598686 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:57.598616 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b916d612-a355-4525-8143-9e3a47ead6d0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b916d612-a355-4525-8143-9e3a47ead6d0-model-cache\") pod \"b916d612-a355-4525-8143-9e3a47ead6d0\" (UID: \"b916d612-a355-4525-8143-9e3a47ead6d0\") " Nov 01 20:24:57.598898 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:57.598695 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a948974-537a-49da-85e7-60adf68d1655-home\" (UniqueName: \"kubernetes.io/empty-dir/4a948974-537a-49da-85e7-60adf68d1655-home\") pod \"4a948974-537a-49da-85e7-60adf68d1655\" (UID: \"4a948974-537a-49da-85e7-60adf68d1655\") " Nov 01 20:24:57.599227 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:57.599202 5935 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4a948974-537a-49da-85e7-60adf68d1655-model-cache\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:24:57.599227 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:57.599229 5935 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b916d612-a355-4525-8143-9e3a47ead6d0-home\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:24:57.599550 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:57.599416 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a948974-537a-49da-85e7-60adf68d1655-tmp-dir" pod "4a948974-537a-49da-85e7-60adf68d1655" (UID: "4a948974-537a-49da-85e7-60adf68d1655"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:24:57.599669 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:57.599648 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b916d612-a355-4525-8143-9e3a47ead6d0-tmp-dir" pod "b916d612-a355-4525-8143-9e3a47ead6d0" (UID: "b916d612-a355-4525-8143-9e3a47ead6d0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:24:57.599741 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:57.599667 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b916d612-a355-4525-8143-9e3a47ead6d0-model-cache" pod "b916d612-a355-4525-8143-9e3a47ead6d0" (UID: "b916d612-a355-4525-8143-9e3a47ead6d0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:24:57.599741 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:57.599736 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a948974-537a-49da-85e7-60adf68d1655-home" pod "4a948974-537a-49da-85e7-60adf68d1655" (UID: "4a948974-537a-49da-85e7-60adf68d1655"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:24:57.601092 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:57.601057 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a948974-537a-49da-85e7-60adf68d1655-dshm" pod "4a948974-537a-49da-85e7-60adf68d1655" (UID: "4a948974-537a-49da-85e7-60adf68d1655"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:24:57.601210 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:57.601165 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a948974-537a-49da-85e7-60adf68d1655-kube-api-access-dqchf" pod "4a948974-537a-49da-85e7-60adf68d1655" (UID: "4a948974-537a-49da-85e7-60adf68d1655"). InnerVolumeSpecName "kube-api-access-dqchf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:24:57.601366 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:57.601344 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b916d612-a355-4525-8143-9e3a47ead6d0-tls-certs" pod "b916d612-a355-4525-8143-9e3a47ead6d0" (UID: "b916d612-a355-4525-8143-9e3a47ead6d0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:24:57.601447 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:57.601399 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b916d612-a355-4525-8143-9e3a47ead6d0-kube-api-access-kv6lk" pod "b916d612-a355-4525-8143-9e3a47ead6d0" (UID: "b916d612-a355-4525-8143-9e3a47ead6d0"). InnerVolumeSpecName "kube-api-access-kv6lk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:24:57.601764 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:57.601736 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b916d612-a355-4525-8143-9e3a47ead6d0-dshm" pod "b916d612-a355-4525-8143-9e3a47ead6d0" (UID: "b916d612-a355-4525-8143-9e3a47ead6d0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:24:57.602740 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:57.602716 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a948974-537a-49da-85e7-60adf68d1655-tls-certs" pod "4a948974-537a-49da-85e7-60adf68d1655" (UID: "4a948974-537a-49da-85e7-60adf68d1655"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:24:57.700750 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:57.700720 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqchf\" (UniqueName: \"kubernetes.io/projected/4a948974-537a-49da-85e7-60adf68d1655-kube-api-access-dqchf\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:24:57.700750 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:57.700748 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kv6lk\" (UniqueName: \"kubernetes.io/projected/b916d612-a355-4525-8143-9e3a47ead6d0-kube-api-access-kv6lk\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:24:57.700941 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:57.700759 5935 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4a948974-537a-49da-85e7-60adf68d1655-tmp-dir\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:24:57.700941 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:57.700768 5935 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4a948974-537a-49da-85e7-60adf68d1655-dshm\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:24:57.700941 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:57.700776 5935 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4a948974-537a-49da-85e7-60adf68d1655-tls-certs\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:24:57.700941 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:57.700785 5935 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b916d612-a355-4525-8143-9e3a47ead6d0-tmp-dir\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:24:57.700941 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:57.700792 5935 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b916d612-a355-4525-8143-9e3a47ead6d0-model-cache\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:24:57.700941 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:57.700801 5935 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4a948974-537a-49da-85e7-60adf68d1655-home\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:24:57.700941 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:57.700810 5935 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b916d612-a355-4525-8143-9e3a47ead6d0-tls-certs\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:24:57.700941 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:57.700819 5935 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b916d612-a355-4525-8143-9e3a47ead6d0-dshm\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:24:58.303903 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:58.303872 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e27ae7/distinct-w2-e27ae7-kserve-7df8547d56-9qgdk" Nov 01 20:24:58.303903 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:58.303895 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e27ae7/distinct-w2-e27ae7-kserve-7df8547d56-9qgdk" event={"ID":"4a948974-537a-49da-85e7-60adf68d1655","Type":"ContainerDied","Data":"07380a329b12a2288e6484aca34c7ed601fd1a906a98b7948828a5ae6e495e75"} Nov 01 20:24:58.304408 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:58.303950 5935 scope.go:122] "RemoveContainer" containerID="cae5df9029be86bfb8b61d6230defafa4ae76e893262625fd835f376329609f6" Nov 01 20:24:58.305348 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:58.305269 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e27ae7/distinct-2-w2-e27ae7-kserve-6bf4d46475-jdjhp" event={"ID":"b916d612-a355-4525-8143-9e3a47ead6d0","Type":"ContainerDied","Data":"28472daaab66c58d9f340e63390add390fe32fd610d058f0ee56e64c3366dc76"} Nov 01 20:24:58.305445 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:58.305393 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e27ae7/distinct-2-w2-e27ae7-kserve-6bf4d46475-jdjhp" Nov 01 20:24:58.315473 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:58.315455 5935 scope.go:122] "RemoveContainer" containerID="0ca1f24ac8ccb1cf5520880d24e4719caf99cc6c1de37d73d8f2239df2d2190d" Nov 01 20:24:58.327192 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:58.327173 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e27ae7/distinct-w2-e27ae7-kserve-7df8547d56-9qgdk"] Nov 01 20:24:58.333896 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:58.333877 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-e27ae7/distinct-w2-e27ae7-kserve-7df8547d56-9qgdk"] Nov 01 20:24:58.342925 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:58.342902 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e27ae7/distinct-2-w2-e27ae7-kserve-6bf4d46475-jdjhp"] Nov 01 20:24:58.345637 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:58.345619 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-e27ae7/distinct-2-w2-e27ae7-kserve-6bf4d46475-jdjhp"] Nov 01 20:24:58.353127 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:58.353107 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a948974-537a-49da-85e7-60adf68d1655" path="/var/lib/kubelet/pods/4a948974-537a-49da-85e7-60adf68d1655/volumes" Nov 01 20:24:58.353667 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:24:58.353651 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b916d612-a355-4525-8143-9e3a47ead6d0" path="/var/lib/kubelet/pods/b916d612-a355-4525-8143-9e3a47ead6d0/volumes" Nov 01 20:25:05.846758 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:25:05.846722 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:25:06.703081 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:06.703037 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e27ae7-7c877fb66c-hglrt" podUID="b7d68b3d-ba87-4c71-9399-989f83e198b4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 20:25:10.037910 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:10.037872 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p"] Nov 01 20:25:10.048962 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:10.047338 5935 cpu_manager.go:414] "containerMap: removing container" podUID="4a948974-537a-49da-85e7-60adf68d1655" containerName="main" Nov 01 20:25:10.052385 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:10.051649 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a948974-537a-49da-85e7-60adf68d1655" containerName="main" Nov 01 20:25:10.066307 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:10.054355 5935 cpu_manager.go:414] "containerMap: removing container" podUID="3cf4dde1-ffc3-4640-8b7f-65f3c4173e47" containerName="main" Nov 01 20:25:10.067019 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:10.066995 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cf4dde1-ffc3-4640-8b7f-65f3c4173e47" containerName="main" Nov 01 20:25:10.070302 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:10.067193 5935 cpu_manager.go:414] "containerMap: removing container" podUID="3b63dca7-1aeb-40c5-8b7c-328578a58adf" containerName="istio-proxy" Nov 01 20:25:10.070443 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:10.070425 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b63dca7-1aeb-40c5-8b7c-328578a58adf" containerName="istio-proxy" Nov 01 20:25:10.070549 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:10.070538 5935 cpu_manager.go:414] "containerMap: removing container" podUID="b916d612-a355-4525-8143-9e3a47ead6d0" containerName="main" Nov 01 20:25:10.118311 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:10.094389 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b916d612-a355-4525-8143-9e3a47ead6d0" containerName="main" Nov 01 20:25:10.199319 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:10.175563 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a948974-537a-49da-85e7-60adf68d1655" containerName="main" Nov 01 20:25:10.199319 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:10.175598 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cf4dde1-ffc3-4640-8b7f-65f3c4173e47" containerName="main" Nov 01 20:25:10.199319 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:10.175619 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b63dca7-1aeb-40c5-8b7c-328578a58adf" containerName="istio-proxy" Nov 01 20:25:10.199319 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:10.175631 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="b916d612-a355-4525-8143-9e3a47ead6d0" containerName="main" Nov 01 20:25:10.284333 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:10.267902 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p"] Nov 01 20:25:10.284333 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:10.268042 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p" Nov 01 20:25:10.336377 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:10.332246 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-9ea5d2-openshift-default-dockercfg-qgnmd\"" Nov 01 20:25:10.454366 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:10.454334 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2babaef6-77b4b5bd9b-k7zzz"] Nov 01 20:25:10.454910 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:10.454848 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2babaef6-77b4b5bd9b-k7zzz" podUID="49e78298-4681-4a39-89b3-65eeae2d3c22" containerName="maas-api" containerID="cri-o://14d9fd31b38825bfc26c680c2dee59d4d9c2efc1fb78caf281b11192d7ec174d" gracePeriod=30 Nov 01 20:25:10.518307 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:10.500902 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0-istiod-ca-cert\") pod \"e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p\" (UID: \"0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0\") " pod="openshift-ingress/e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p" Nov 01 20:25:10.518307 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:10.516208 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0-workload-socket\") pod \"e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p\" (UID: \"0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0\") " pod="openshift-ingress/e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p" Nov 01 20:25:10.518307 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:10.516264 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9cpb\" (UniqueName: \"kubernetes.io/projected/0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0-kube-api-access-k9cpb\") pod \"e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p\" (UID: \"0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0\") " pod="openshift-ingress/e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p" Nov 01 20:25:10.518307 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:10.516327 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0-istio-podinfo\") pod \"e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p\" (UID: \"0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0\") " pod="openshift-ingress/e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p" Nov 01 20:25:10.518307 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:10.516389 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0-istio-data\") pod \"e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p\" (UID: \"0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0\") " pod="openshift-ingress/e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p" Nov 01 20:25:10.518307 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:10.516422 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0-istio-token\") pod \"e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p\" (UID: \"0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0\") " pod="openshift-ingress/e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p" Nov 01 20:25:10.518307 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:10.516505 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0-credential-socket\") pod \"e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p\" (UID: \"0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0\") " pod="openshift-ingress/e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p" Nov 01 20:25:10.518307 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:10.516576 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0-workload-certs\") pod \"e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p\" (UID: \"0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0\") " pod="openshift-ingress/e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p" Nov 01 20:25:10.518307 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:10.516616 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0-istio-envoy\") pod \"e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p\" (UID: \"0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0\") " pod="openshift-ingress/e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p" Nov 01 20:25:10.668681 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:10.660063 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0-istiod-ca-cert\") pod \"e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p\" (UID: \"0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0\") " pod="openshift-ingress/e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p" Nov 01 20:25:10.775590 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:10.703409 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0-workload-socket\") pod \"e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p\" (UID: \"0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0\") " pod="openshift-ingress/e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p" Nov 01 20:25:10.775590 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:10.703476 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9cpb\" (UniqueName: \"kubernetes.io/projected/0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0-kube-api-access-k9cpb\") pod \"e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p\" (UID: \"0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0\") " pod="openshift-ingress/e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p" Nov 01 20:25:10.775590 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:10.703523 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0-istio-podinfo\") pod \"e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p\" (UID: \"0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0\") " pod="openshift-ingress/e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p" Nov 01 20:25:10.775590 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:10.703593 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0-istio-data\") pod \"e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p\" (UID: \"0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0\") " pod="openshift-ingress/e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p" Nov 01 20:25:10.775590 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:10.703628 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0-istio-token\") pod \"e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p\" (UID: \"0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0\") " pod="openshift-ingress/e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p" Nov 01 20:25:10.775590 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:10.703726 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0-credential-socket\") pod \"e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p\" (UID: \"0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0\") " pod="openshift-ingress/e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p" Nov 01 20:25:10.775590 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:10.703808 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0-workload-certs\") pod \"e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p\" (UID: \"0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0\") " pod="openshift-ingress/e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p" Nov 01 20:25:10.775590 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:10.703854 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0-istio-envoy\") pod \"e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p\" (UID: \"0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0\") " pod="openshift-ingress/e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p" Nov 01 20:25:10.775590 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:10.668079 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0-istiod-ca-cert\") pod \"e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p\" (UID: \"0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0\") " pod="openshift-ingress/e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p" Nov 01 20:25:10.775590 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:10.704803 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0-workload-socket\") pod \"e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p\" (UID: \"0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0\") " pod="openshift-ingress/e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p" Nov 01 20:25:10.775590 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:10.707202 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0-workload-certs\") pod \"e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p\" (UID: \"0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0\") " pod="openshift-ingress/e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p" Nov 01 20:25:10.775590 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:10.707983 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0-credential-socket\") pod \"e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p\" (UID: \"0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0\") " pod="openshift-ingress/e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p" Nov 01 20:25:10.775590 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:10.708238 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0-istio-data\") pod \"e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p\" (UID: \"0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0\") " pod="openshift-ingress/e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p" Nov 01 20:25:10.799375 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:10.799279 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9cpb\" (UniqueName: \"kubernetes.io/projected/0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0-kube-api-access-k9cpb\") pod \"e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p\" (UID: \"0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0\") " pod="openshift-ingress/e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p" Nov 01 20:25:10.800104 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:10.800050 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0-istio-podinfo\") pod \"e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p\" (UID: \"0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0\") " pod="openshift-ingress/e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p" Nov 01 20:25:10.812319 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:10.811084 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0-istio-token\") pod \"e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p\" (UID: \"0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0\") " pod="openshift-ingress/e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p" Nov 01 20:25:10.813613 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:10.813587 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0-istio-envoy\") pod \"e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p\" (UID: \"0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0\") " pod="openshift-ingress/e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p" Nov 01 20:25:10.981154 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:10.981123 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p" Nov 01 20:25:11.059992 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:11.059964 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2babaef6-77b4b5bd9b-k7zzz" Nov 01 20:25:11.110031 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:11.108859 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49e78298-4681-4a39-89b3-65eeae2d3c22-metrics-certs\" (UniqueName: \"kubernetes.io/secret/49e78298-4681-4a39-89b3-65eeae2d3c22-metrics-certs\") pod \"49e78298-4681-4a39-89b3-65eeae2d3c22\" (UID: \"49e78298-4681-4a39-89b3-65eeae2d3c22\") " Nov 01 20:25:11.110031 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:11.108986 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/49e78298-4681-4a39-89b3-65eeae2d3c22-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49e78298-4681-4a39-89b3-65eeae2d3c22-maas-api-tls\") pod \"49e78298-4681-4a39-89b3-65eeae2d3c22\" (UID: \"49e78298-4681-4a39-89b3-65eeae2d3c22\") " Nov 01 20:25:11.110031 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:11.109058 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49e78298-4681-4a39-89b3-65eeae2d3c22-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49e78298-4681-4a39-89b3-65eeae2d3c22-openshift-service-ca\") pod \"49e78298-4681-4a39-89b3-65eeae2d3c22\" (UID: \"49e78298-4681-4a39-89b3-65eeae2d3c22\") " Nov 01 20:25:11.110031 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:11.109087 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49e78298-4681-4a39-89b3-65eeae2d3c22-kube-api-access-t9s7v\" (UniqueName: \"kubernetes.io/projected/49e78298-4681-4a39-89b3-65eeae2d3c22-kube-api-access-t9s7v\") pod \"49e78298-4681-4a39-89b3-65eeae2d3c22\" (UID: \"49e78298-4681-4a39-89b3-65eeae2d3c22\") " Nov 01 20:25:11.110775 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:11.110749 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49e78298-4681-4a39-89b3-65eeae2d3c22-openshift-service-ca" pod "49e78298-4681-4a39-89b3-65eeae2d3c22" (UID: "49e78298-4681-4a39-89b3-65eeae2d3c22"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:25:11.112838 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:11.112800 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49e78298-4681-4a39-89b3-65eeae2d3c22-maas-api-tls" pod "49e78298-4681-4a39-89b3-65eeae2d3c22" (UID: "49e78298-4681-4a39-89b3-65eeae2d3c22"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:25:11.112945 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:11.112838 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49e78298-4681-4a39-89b3-65eeae2d3c22-kube-api-access-t9s7v" pod "49e78298-4681-4a39-89b3-65eeae2d3c22" (UID: "49e78298-4681-4a39-89b3-65eeae2d3c22"). InnerVolumeSpecName "kube-api-access-t9s7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:25:11.113691 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:11.113666 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49e78298-4681-4a39-89b3-65eeae2d3c22-metrics-certs" pod "49e78298-4681-4a39-89b3-65eeae2d3c22" (UID: "49e78298-4681-4a39-89b3-65eeae2d3c22"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:25:11.212075 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:11.212023 5935 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/49e78298-4681-4a39-89b3-65eeae2d3c22-metrics-certs\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:25:11.212075 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:11.212062 5935 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/49e78298-4681-4a39-89b3-65eeae2d3c22-maas-api-tls\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:25:11.212075 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:11.212080 5935 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/49e78298-4681-4a39-89b3-65eeae2d3c22-openshift-service-ca\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:25:11.212446 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:11.212095 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9s7v\" (UniqueName: \"kubernetes.io/projected/49e78298-4681-4a39-89b3-65eeae2d3c22-kube-api-access-t9s7v\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:25:11.315609 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:11.315577 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p"] Nov 01 20:25:11.318197 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:25:11.318158 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a36a3c4_57f6_4486_9adb_0ef1af2fb3d0.slice/crio-8f0ee3006d566e49cba910528569f8baab86691911006f75c1da310a5ef015e8 WatchSource:0}: Error finding container 8f0ee3006d566e49cba910528569f8baab86691911006f75c1da310a5ef015e8: Status 404 returned error can't find the container with id 8f0ee3006d566e49cba910528569f8baab86691911006f75c1da310a5ef015e8 Nov 01 20:25:11.320866 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:11.320828 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:25:11.320990 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:11.320903 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:25:11.320990 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:11.320933 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:25:11.510374 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:11.510246 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p" event={"ID":"0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0","Type":"ContainerStarted","Data":"9e0b2bf5d91d0d6af1a2f8176065780be9ccb24b68e00573db4a0ffb1c2c086a"} Nov 01 20:25:11.510374 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:11.510329 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p" event={"ID":"0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0","Type":"ContainerStarted","Data":"8f0ee3006d566e49cba910528569f8baab86691911006f75c1da310a5ef015e8"} Nov 01 20:25:11.512327 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:11.512266 5935 generic.go:356] "Generic (PLEG): container finished" podID="49e78298-4681-4a39-89b3-65eeae2d3c22" containerID="14d9fd31b38825bfc26c680c2dee59d4d9c2efc1fb78caf281b11192d7ec174d" exitCode=0 Nov 01 20:25:11.512467 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:11.512344 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2babaef6-77b4b5bd9b-k7zzz" event={"ID":"49e78298-4681-4a39-89b3-65eeae2d3c22","Type":"ContainerDied","Data":"14d9fd31b38825bfc26c680c2dee59d4d9c2efc1fb78caf281b11192d7ec174d"} Nov 01 20:25:11.512467 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:11.512384 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2babaef6-77b4b5bd9b-k7zzz" event={"ID":"49e78298-4681-4a39-89b3-65eeae2d3c22","Type":"ContainerDied","Data":"fa6c2969274c61dc51b9a0c8129778bfc0d2381ac029da0853abe95bffd9592f"} Nov 01 20:25:11.512467 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:11.512412 5935 scope.go:122] "RemoveContainer" containerID="14d9fd31b38825bfc26c680c2dee59d4d9c2efc1fb78caf281b11192d7ec174d" Nov 01 20:25:11.512625 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:11.512562 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2babaef6-77b4b5bd9b-k7zzz" Nov 01 20:25:11.532675 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:11.532623 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p" podStartSLOduration=2.532604516 podStartE2EDuration="2.532604516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:25:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:25:11.528504664 +0000 UTC m=+1557.301665177" watchObservedRunningTime="2026-11-01 20:25:11.532604516 +0000 UTC m=+1557.305765027" Nov 01 20:25:11.534486 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:11.534464 5935 scope.go:122] "RemoveContainer" containerID="14d9fd31b38825bfc26c680c2dee59d4d9c2efc1fb78caf281b11192d7ec174d" Nov 01 20:25:11.535039 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:25:11.535011 5935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14d9fd31b38825bfc26c680c2dee59d4d9c2efc1fb78caf281b11192d7ec174d\": container with ID starting with 14d9fd31b38825bfc26c680c2dee59d4d9c2efc1fb78caf281b11192d7ec174d not found: ID does not exist" containerID="14d9fd31b38825bfc26c680c2dee59d4d9c2efc1fb78caf281b11192d7ec174d" Nov 01 20:25:11.535151 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:11.535052 5935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14d9fd31b38825bfc26c680c2dee59d4d9c2efc1fb78caf281b11192d7ec174d"} err="failed to get container status \"14d9fd31b38825bfc26c680c2dee59d4d9c2efc1fb78caf281b11192d7ec174d\": rpc error: code = NotFound desc = could not find container \"14d9fd31b38825bfc26c680c2dee59d4d9c2efc1fb78caf281b11192d7ec174d\": container with ID starting with 14d9fd31b38825bfc26c680c2dee59d4d9c2efc1fb78caf281b11192d7ec174d not found: ID does not exist" Nov 01 20:25:11.552350 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:11.552322 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2babaef6-77b4b5bd9b-k7zzz"] Nov 01 20:25:11.552485 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:11.552445 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2babaef6-77b4b5bd9b-k7zzz"] Nov 01 20:25:11.981265 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:11.981225 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p" Nov 01 20:25:12.355766 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:12.355693 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49e78298-4681-4a39-89b3-65eeae2d3c22" path="/var/lib/kubelet/pods/49e78298-4681-4a39-89b3-65eeae2d3c22/volumes" Nov 01 20:25:12.991188 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:12.991155 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p" Nov 01 20:25:13.421050 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:13.420938 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-e27ae7-8595cd7889-ttqzl"] Nov 01 20:25:13.421618 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:13.421167 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e27ae7-8595cd7889-ttqzl" podUID="77d9168b-ce7b-47ce-999c-9f9622472d48" containerName="maas-api" containerID="cri-o://d7cd6a447ac638d560f77bf4981c73ac785af399b2acd11214872c4669521d9a" gracePeriod=30 Nov 01 20:25:13.530824 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:13.530787 5935 generic.go:356] "Generic (PLEG): container finished" podID="b7d68b3d-ba87-4c71-9399-989f83e198b4" containerID="a8d6eee555e6d0a20c087b0b50848246eb6b8dcb0b4dc2f752041511e25246fc" exitCode=0 Nov 01 20:25:13.531011 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:13.530870 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e27ae7-7c877fb66c-hglrt" event={"ID":"b7d68b3d-ba87-4c71-9399-989f83e198b4","Type":"ContainerDied","Data":"a8d6eee555e6d0a20c087b0b50848246eb6b8dcb0b4dc2f752041511e25246fc"} Nov 01 20:25:13.533363 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:13.531260 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p" Nov 01 20:25:13.533363 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:13.532654 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p" Nov 01 20:25:13.706483 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:13.705311 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e27ae7-7c877fb66c-hglrt" Nov 01 20:25:13.744924 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:13.744893 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7d68b3d-ba87-4c71-9399-989f83e198b4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7d68b3d-ba87-4c71-9399-989f83e198b4-plugins-config-volume\") pod \"b7d68b3d-ba87-4c71-9399-989f83e198b4\" (UID: \"b7d68b3d-ba87-4c71-9399-989f83e198b4\") " Nov 01 20:25:13.745092 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:13.745066 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7d68b3d-ba87-4c71-9399-989f83e198b4-kube-api-access-wwlpw\" (UniqueName: \"kubernetes.io/projected/b7d68b3d-ba87-4c71-9399-989f83e198b4-kube-api-access-wwlpw\") pod \"b7d68b3d-ba87-4c71-9399-989f83e198b4\" (UID: \"b7d68b3d-ba87-4c71-9399-989f83e198b4\") " Nov 01 20:25:13.745347 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:13.745313 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7d68b3d-ba87-4c71-9399-989f83e198b4-plugins-config-volume" pod "b7d68b3d-ba87-4c71-9399-989f83e198b4" (UID: "b7d68b3d-ba87-4c71-9399-989f83e198b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:25:13.745543 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:13.745526 5935 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7d68b3d-ba87-4c71-9399-989f83e198b4-plugins-config-volume\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:25:13.747425 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:13.747388 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7d68b3d-ba87-4c71-9399-989f83e198b4-kube-api-access-wwlpw" pod "b7d68b3d-ba87-4c71-9399-989f83e198b4" (UID: "b7d68b3d-ba87-4c71-9399-989f83e198b4"). InnerVolumeSpecName "kube-api-access-wwlpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:25:13.847313 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:13.846506 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwlpw\" (UniqueName: \"kubernetes.io/projected/b7d68b3d-ba87-4c71-9399-989f83e198b4-kube-api-access-wwlpw\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:25:13.898894 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:13.898868 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e27ae7-8595cd7889-ttqzl" Nov 01 20:25:13.962360 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:13.955441 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77d9168b-ce7b-47ce-999c-9f9622472d48-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77d9168b-ce7b-47ce-999c-9f9622472d48-maas-api-tls\") pod \"77d9168b-ce7b-47ce-999c-9f9622472d48\" (UID: \"77d9168b-ce7b-47ce-999c-9f9622472d48\") " Nov 01 20:25:13.962360 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:13.955609 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77d9168b-ce7b-47ce-999c-9f9622472d48-metrics-certs\" (UniqueName: \"kubernetes.io/secret/77d9168b-ce7b-47ce-999c-9f9622472d48-metrics-certs\") pod \"77d9168b-ce7b-47ce-999c-9f9622472d48\" (UID: \"77d9168b-ce7b-47ce-999c-9f9622472d48\") " Nov 01 20:25:13.962360 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:13.955675 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77d9168b-ce7b-47ce-999c-9f9622472d48-kube-api-access-ftpzt\" (UniqueName: \"kubernetes.io/projected/77d9168b-ce7b-47ce-999c-9f9622472d48-kube-api-access-ftpzt\") pod \"77d9168b-ce7b-47ce-999c-9f9622472d48\" (UID: \"77d9168b-ce7b-47ce-999c-9f9622472d48\") " Nov 01 20:25:13.962360 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:13.955706 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77d9168b-ce7b-47ce-999c-9f9622472d48-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77d9168b-ce7b-47ce-999c-9f9622472d48-openshift-service-ca\") pod \"77d9168b-ce7b-47ce-999c-9f9622472d48\" (UID: \"77d9168b-ce7b-47ce-999c-9f9622472d48\") " Nov 01 20:25:13.962360 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:13.957785 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77d9168b-ce7b-47ce-999c-9f9622472d48-openshift-service-ca" pod "77d9168b-ce7b-47ce-999c-9f9622472d48" (UID: "77d9168b-ce7b-47ce-999c-9f9622472d48"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:25:13.962360 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:13.959890 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77d9168b-ce7b-47ce-999c-9f9622472d48-maas-api-tls" pod "77d9168b-ce7b-47ce-999c-9f9622472d48" (UID: "77d9168b-ce7b-47ce-999c-9f9622472d48"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:25:13.965100 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:13.963806 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77d9168b-ce7b-47ce-999c-9f9622472d48-kube-api-access-ftpzt" pod "77d9168b-ce7b-47ce-999c-9f9622472d48" (UID: "77d9168b-ce7b-47ce-999c-9f9622472d48"). InnerVolumeSpecName "kube-api-access-ftpzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:25:13.965100 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:13.965094 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77d9168b-ce7b-47ce-999c-9f9622472d48-metrics-certs" pod "77d9168b-ce7b-47ce-999c-9f9622472d48" (UID: "77d9168b-ce7b-47ce-999c-9f9622472d48"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:25:14.058834 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:14.058801 5935 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/77d9168b-ce7b-47ce-999c-9f9622472d48-maas-api-tls\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:25:14.058834 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:14.058829 5935 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/77d9168b-ce7b-47ce-999c-9f9622472d48-metrics-certs\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:25:14.059031 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:14.058843 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftpzt\" (UniqueName: \"kubernetes.io/projected/77d9168b-ce7b-47ce-999c-9f9622472d48-kube-api-access-ftpzt\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:25:14.059031 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:14.058857 5935 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/77d9168b-ce7b-47ce-999c-9f9622472d48-openshift-service-ca\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:25:14.538452 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:14.538425 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e27ae7-7c877fb66c-hglrt" Nov 01 20:25:14.538886 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:14.538426 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e27ae7-7c877fb66c-hglrt" event={"ID":"b7d68b3d-ba87-4c71-9399-989f83e198b4","Type":"ContainerDied","Data":"c7df082b1e1d8ea5c9e9d9c8206dad35f65873faeb5db27c4d352fde9a929dbe"} Nov 01 20:25:14.538886 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:14.538559 5935 scope.go:122] "RemoveContainer" containerID="a8d6eee555e6d0a20c087b0b50848246eb6b8dcb0b4dc2f752041511e25246fc" Nov 01 20:25:14.540125 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:14.540103 5935 generic.go:356] "Generic (PLEG): container finished" podID="77d9168b-ce7b-47ce-999c-9f9622472d48" containerID="d7cd6a447ac638d560f77bf4981c73ac785af399b2acd11214872c4669521d9a" exitCode=0 Nov 01 20:25:14.540230 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:14.540145 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e27ae7-8595cd7889-ttqzl" Nov 01 20:25:14.540230 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:14.540193 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e27ae7-8595cd7889-ttqzl" event={"ID":"77d9168b-ce7b-47ce-999c-9f9622472d48","Type":"ContainerDied","Data":"d7cd6a447ac638d560f77bf4981c73ac785af399b2acd11214872c4669521d9a"} Nov 01 20:25:14.540341 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:14.540247 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e27ae7-8595cd7889-ttqzl" event={"ID":"77d9168b-ce7b-47ce-999c-9f9622472d48","Type":"ContainerDied","Data":"56dd5b98dd721405c00f63c304332f060f12d8cbb249be04e18ffba66e836d0f"} Nov 01 20:25:14.549929 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:14.549913 5935 scope.go:122] "RemoveContainer" containerID="d7cd6a447ac638d560f77bf4981c73ac785af399b2acd11214872c4669521d9a" Nov 01 20:25:14.558676 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:14.558660 5935 scope.go:122] "RemoveContainer" containerID="d7cd6a447ac638d560f77bf4981c73ac785af399b2acd11214872c4669521d9a" Nov 01 20:25:14.558929 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:25:14.558908 5935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7cd6a447ac638d560f77bf4981c73ac785af399b2acd11214872c4669521d9a\": container with ID starting with d7cd6a447ac638d560f77bf4981c73ac785af399b2acd11214872c4669521d9a not found: ID does not exist" containerID="d7cd6a447ac638d560f77bf4981c73ac785af399b2acd11214872c4669521d9a" Nov 01 20:25:14.558972 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:14.558939 5935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7cd6a447ac638d560f77bf4981c73ac785af399b2acd11214872c4669521d9a"} err="failed to get container status \"d7cd6a447ac638d560f77bf4981c73ac785af399b2acd11214872c4669521d9a\": rpc error: code = NotFound desc = could not find container \"d7cd6a447ac638d560f77bf4981c73ac785af399b2acd11214872c4669521d9a\": container with ID starting with d7cd6a447ac638d560f77bf4981c73ac785af399b2acd11214872c4669521d9a not found: ID does not exist" Nov 01 20:25:14.567797 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:14.567775 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-e27ae7-8595cd7889-ttqzl"] Nov 01 20:25:14.570492 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:14.570470 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-e27ae7-8595cd7889-ttqzl"] Nov 01 20:25:14.590720 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:14.590696 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-e27ae7-7c877fb66c-hglrt"] Nov 01 20:25:14.593100 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:14.593078 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-e27ae7-7c877fb66c-hglrt"] Nov 01 20:25:15.087538 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:15.087474 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-9ea5d2-78b67f847f-j8dj9"] Nov 01 20:25:15.088017 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:15.088005 5935 cpu_manager.go:414] "containerMap: removing container" podUID="b7d68b3d-ba87-4c71-9399-989f83e198b4" containerName="payload-pre-processing" Nov 01 20:25:15.088064 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:15.088019 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7d68b3d-ba87-4c71-9399-989f83e198b4" containerName="payload-pre-processing" Nov 01 20:25:15.088100 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:15.088080 5935 cpu_manager.go:414] "containerMap: removing container" podUID="77d9168b-ce7b-47ce-999c-9f9622472d48" containerName="maas-api" Nov 01 20:25:15.088100 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:15.088089 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77d9168b-ce7b-47ce-999c-9f9622472d48" containerName="maas-api" Nov 01 20:25:15.088100 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:15.088098 5935 cpu_manager.go:414] "containerMap: removing container" podUID="49e78298-4681-4a39-89b3-65eeae2d3c22" containerName="maas-api" Nov 01 20:25:15.088202 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:15.088109 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49e78298-4681-4a39-89b3-65eeae2d3c22" containerName="maas-api" Nov 01 20:25:15.088258 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:15.088236 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="77d9168b-ce7b-47ce-999c-9f9622472d48" containerName="maas-api" Nov 01 20:25:15.088258 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:15.088255 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="49e78298-4681-4a39-89b3-65eeae2d3c22" containerName="maas-api" Nov 01 20:25:15.088395 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:15.088271 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7d68b3d-ba87-4c71-9399-989f83e198b4" containerName="payload-pre-processing" Nov 01 20:25:15.093462 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:15.093440 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9ea5d2-78b67f847f-j8dj9" Nov 01 20:25:15.103947 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:15.102258 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-9ea5d2\"" Nov 01 20:25:15.103947 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:15.102358 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-9ea5d2-dockercfg-flwq4\"" Nov 01 20:25:15.127662 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:15.127631 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-9ea5d2-78b67f847f-j8dj9"] Nov 01 20:25:15.138276 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:15.138252 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-9ea5d2-6c96fd4f45-lxrs4"] Nov 01 20:25:15.142199 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:15.142181 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-9ea5d2-6c96fd4f45-lxrs4" Nov 01 20:25:15.171759 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:15.171713 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49ab9daf-af2a-452e-9860-24b64352ca82-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-9ea5d2-78b67f847f-j8dj9\" (UID: \"49ab9daf-af2a-452e-9860-24b64352ca82\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9ea5d2-78b67f847f-j8dj9" Nov 01 20:25:15.172376 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:15.172352 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvd6z\" (UniqueName: \"kubernetes.io/projected/49ab9daf-af2a-452e-9860-24b64352ca82-kube-api-access-gvd6z\") pod \"payload-pre-processing-e2e-extmod-b-9ea5d2-78b67f847f-j8dj9\" (UID: \"49ab9daf-af2a-452e-9860-24b64352ca82\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9ea5d2-78b67f847f-j8dj9" Nov 01 20:25:15.191328 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:15.182090 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-9ea5d2-6c96fd4f45-lxrs4"] Nov 01 20:25:15.274130 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:15.274096 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b4bf215-ad37-4389-b71b-e56de6363784-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-9ea5d2-6c96fd4f45-lxrs4\" (UID: \"3b4bf215-ad37-4389-b71b-e56de6363784\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-9ea5d2-6c96fd4f45-lxrs4" Nov 01 20:25:15.274784 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:15.274386 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvd6z\" (UniqueName: \"kubernetes.io/projected/49ab9daf-af2a-452e-9860-24b64352ca82-kube-api-access-gvd6z\") pod \"payload-pre-processing-e2e-extmod-b-9ea5d2-78b67f847f-j8dj9\" (UID: \"49ab9daf-af2a-452e-9860-24b64352ca82\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9ea5d2-78b67f847f-j8dj9" Nov 01 20:25:15.274784 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:15.274445 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-794xj\" (UniqueName: \"kubernetes.io/projected/3b4bf215-ad37-4389-b71b-e56de6363784-kube-api-access-794xj\") pod \"payload-processing-e2e-extmod-b-9ea5d2-6c96fd4f45-lxrs4\" (UID: \"3b4bf215-ad37-4389-b71b-e56de6363784\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-9ea5d2-6c96fd4f45-lxrs4" Nov 01 20:25:15.274784 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:15.274490 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49ab9daf-af2a-452e-9860-24b64352ca82-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-9ea5d2-78b67f847f-j8dj9\" (UID: \"49ab9daf-af2a-452e-9860-24b64352ca82\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9ea5d2-78b67f847f-j8dj9" Nov 01 20:25:15.275404 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:15.275380 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49ab9daf-af2a-452e-9860-24b64352ca82-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-9ea5d2-78b67f847f-j8dj9\" (UID: \"49ab9daf-af2a-452e-9860-24b64352ca82\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9ea5d2-78b67f847f-j8dj9" Nov 01 20:25:15.282506 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:15.282486 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvd6z\" (UniqueName: \"kubernetes.io/projected/49ab9daf-af2a-452e-9860-24b64352ca82-kube-api-access-gvd6z\") pod \"payload-pre-processing-e2e-extmod-b-9ea5d2-78b67f847f-j8dj9\" (UID: \"49ab9daf-af2a-452e-9860-24b64352ca82\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9ea5d2-78b67f847f-j8dj9" Nov 01 20:25:15.375345 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:15.375257 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-794xj\" (UniqueName: \"kubernetes.io/projected/3b4bf215-ad37-4389-b71b-e56de6363784-kube-api-access-794xj\") pod \"payload-processing-e2e-extmod-b-9ea5d2-6c96fd4f45-lxrs4\" (UID: \"3b4bf215-ad37-4389-b71b-e56de6363784\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-9ea5d2-6c96fd4f45-lxrs4" Nov 01 20:25:15.375490 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:15.375442 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b4bf215-ad37-4389-b71b-e56de6363784-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-9ea5d2-6c96fd4f45-lxrs4\" (UID: \"3b4bf215-ad37-4389-b71b-e56de6363784\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-9ea5d2-6c96fd4f45-lxrs4" Nov 01 20:25:15.376170 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:15.376151 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b4bf215-ad37-4389-b71b-e56de6363784-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-9ea5d2-6c96fd4f45-lxrs4\" (UID: \"3b4bf215-ad37-4389-b71b-e56de6363784\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-9ea5d2-6c96fd4f45-lxrs4" Nov 01 20:25:15.384151 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:15.384127 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-794xj\" (UniqueName: \"kubernetes.io/projected/3b4bf215-ad37-4389-b71b-e56de6363784-kube-api-access-794xj\") pod \"payload-processing-e2e-extmod-b-9ea5d2-6c96fd4f45-lxrs4\" (UID: \"3b4bf215-ad37-4389-b71b-e56de6363784\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-9ea5d2-6c96fd4f45-lxrs4" Nov 01 20:25:15.412901 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:15.412871 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9ea5d2-78b67f847f-j8dj9" Nov 01 20:25:15.461613 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:15.461575 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-9ea5d2-6c96fd4f45-lxrs4" Nov 01 20:25:15.705255 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:15.705217 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-9ea5d2-78b67f847f-j8dj9"] Nov 01 20:25:15.706701 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:25:15.706676 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49ab9daf_af2a_452e_9860_24b64352ca82.slice/crio-e7113e6207259d97602348d4dd988c8d6a75a30a7b6010b89c4c5300586a1d5d WatchSource:0}: Error finding container e7113e6207259d97602348d4dd988c8d6a75a30a7b6010b89c4c5300586a1d5d: Status 404 returned error can't find the container with id e7113e6207259d97602348d4dd988c8d6a75a30a7b6010b89c4c5300586a1d5d Nov 01 20:25:15.770761 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:15.770724 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-9ea5d2-6c96fd4f45-lxrs4"] Nov 01 20:25:15.771504 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:25:15.771478 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b4bf215_ad37_4389_b71b_e56de6363784.slice/crio-0d9b4fe1473d6e4695c71667e2eb98f6bae84d0ce079a2b6f140ab40fb1bb384 WatchSource:0}: Error finding container 0d9b4fe1473d6e4695c71667e2eb98f6bae84d0ce079a2b6f140ab40fb1bb384: Status 404 returned error can't find the container with id 0d9b4fe1473d6e4695c71667e2eb98f6bae84d0ce079a2b6f140ab40fb1bb384 Nov 01 20:25:16.354615 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:16.354583 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77d9168b-ce7b-47ce-999c-9f9622472d48" path="/var/lib/kubelet/pods/77d9168b-ce7b-47ce-999c-9f9622472d48/volumes" Nov 01 20:25:16.354973 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:16.354960 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7d68b3d-ba87-4c71-9399-989f83e198b4" path="/var/lib/kubelet/pods/b7d68b3d-ba87-4c71-9399-989f83e198b4/volumes" Nov 01 20:25:16.561970 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:16.561932 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-9ea5d2-6c96fd4f45-lxrs4" event={"ID":"3b4bf215-ad37-4389-b71b-e56de6363784","Type":"ContainerStarted","Data":"49160cc02e2c5108402746b3b2a23508f7ee5c9d001d0c73d6827f34819c1827"} Nov 01 20:25:16.562141 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:16.561994 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-9ea5d2-6c96fd4f45-lxrs4" event={"ID":"3b4bf215-ad37-4389-b71b-e56de6363784","Type":"ContainerStarted","Data":"0d9b4fe1473d6e4695c71667e2eb98f6bae84d0ce079a2b6f140ab40fb1bb384"} Nov 01 20:25:16.562141 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:16.562026 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-9ea5d2-6c96fd4f45-lxrs4" Nov 01 20:25:16.563592 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:16.563566 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9ea5d2-78b67f847f-j8dj9" event={"ID":"49ab9daf-af2a-452e-9860-24b64352ca82","Type":"ContainerStarted","Data":"ede909089d65f9ab2435df8571018bf4dcb887ce3a7860c0838744d234505440"} Nov 01 20:25:16.563707 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:16.563606 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9ea5d2-78b67f847f-j8dj9" event={"ID":"49ab9daf-af2a-452e-9860-24b64352ca82","Type":"ContainerStarted","Data":"e7113e6207259d97602348d4dd988c8d6a75a30a7b6010b89c4c5300586a1d5d"} Nov 01 20:25:16.563707 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:16.563625 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9ea5d2-78b67f847f-j8dj9" Nov 01 20:25:16.578752 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:16.578708 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-9ea5d2-6c96fd4f45-lxrs4" podStartSLOduration=1.578694898 podStartE2EDuration="1.578694898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:25:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:25:16.577064146 +0000 UTC m=+1562.350224670" watchObservedRunningTime="2026-11-01 20:25:16.578694898 +0000 UTC m=+1562.351855410" Nov 01 20:25:16.607515 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:16.607436 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9ea5d2-78b67f847f-j8dj9" podStartSLOduration=1.6074229309999999 podStartE2EDuration="1.607422931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:25:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:25:16.605173895 +0000 UTC m=+1562.378334427" watchObservedRunningTime="2026-11-01 20:25:16.607422931 +0000 UTC m=+1562.380583443" Nov 01 20:25:21.368589 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:21.368558 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6"] Nov 01 20:25:21.369051 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:21.368780 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6" podUID="86e63000-fc7e-4429-af13-4046198bca02" containerName="istio-proxy" containerID="cri-o://12e7c59c07a00198c5a91bebc1dadc8aeef05856ca87046334c3ed2b97dbab6d" gracePeriod=30 Nov 01 20:25:21.372440 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:21.372419 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb"] Nov 01 20:25:21.372821 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:21.372792 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb" podUID="f8249130-8b25-468a-9904-2564d38fdb15" containerName="istio-proxy" containerID="cri-o://c74aa07d56563c2668fc0af4c188dabcff2d8a9a247f4dd9ec20736493faac6a" gracePeriod=30 Nov 01 20:25:21.375914 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:21.375889 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl"] Nov 01 20:25:21.376106 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:21.376081 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl" podUID="16f53549-0a34-4825-a719-7ce0eff818ac" containerName="istio-proxy" containerID="cri-o://98a95c1f612cecb0a0ebe77340028fafeae5d7796fb3e419d9927fbc1491b62a" gracePeriod=30 Nov 01 20:25:24.201866 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:24.201827 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2babaef6-77b4b5bd9b-rzgmf"] Nov 01 20:25:24.207425 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:24.207399 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2babaef6-77b4b5bd9b-rzgmf" Nov 01 20:25:24.216462 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:24.215805 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2babaef6-77b4b5bd9b-rzgmf"] Nov 01 20:25:24.216462 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:24.216235 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-2babaef6-serving-cert\"" Nov 01 20:25:24.329748 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:24.329715 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f89894d-a27e-44f9-8e7a-f1c909500cd4-maas-api-tls\") pod \"maas-api-e2e-mt-2babaef6-77b4b5bd9b-rzgmf\" (UID: \"8f89894d-a27e-44f9-8e7a-f1c909500cd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2babaef6-77b4b5bd9b-rzgmf" Nov 01 20:25:24.329930 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:24.329805 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhsqg\" (UniqueName: \"kubernetes.io/projected/8f89894d-a27e-44f9-8e7a-f1c909500cd4-kube-api-access-xhsqg\") pod \"maas-api-e2e-mt-2babaef6-77b4b5bd9b-rzgmf\" (UID: \"8f89894d-a27e-44f9-8e7a-f1c909500cd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2babaef6-77b4b5bd9b-rzgmf" Nov 01 20:25:24.329930 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:24.329858 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f89894d-a27e-44f9-8e7a-f1c909500cd4-metrics-certs\") pod \"maas-api-e2e-mt-2babaef6-77b4b5bd9b-rzgmf\" (UID: \"8f89894d-a27e-44f9-8e7a-f1c909500cd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2babaef6-77b4b5bd9b-rzgmf" Nov 01 20:25:24.330053 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:24.330038 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f89894d-a27e-44f9-8e7a-f1c909500cd4-openshift-service-ca\") pod \"maas-api-e2e-mt-2babaef6-77b4b5bd9b-rzgmf\" (UID: \"8f89894d-a27e-44f9-8e7a-f1c909500cd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2babaef6-77b4b5bd9b-rzgmf" Nov 01 20:25:24.431427 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:24.431393 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f89894d-a27e-44f9-8e7a-f1c909500cd4-openshift-service-ca\") pod \"maas-api-e2e-mt-2babaef6-77b4b5bd9b-rzgmf\" (UID: \"8f89894d-a27e-44f9-8e7a-f1c909500cd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2babaef6-77b4b5bd9b-rzgmf" Nov 01 20:25:24.431632 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:24.431481 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f89894d-a27e-44f9-8e7a-f1c909500cd4-maas-api-tls\") pod \"maas-api-e2e-mt-2babaef6-77b4b5bd9b-rzgmf\" (UID: \"8f89894d-a27e-44f9-8e7a-f1c909500cd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2babaef6-77b4b5bd9b-rzgmf" Nov 01 20:25:24.431632 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:24.431508 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhsqg\" (UniqueName: \"kubernetes.io/projected/8f89894d-a27e-44f9-8e7a-f1c909500cd4-kube-api-access-xhsqg\") pod \"maas-api-e2e-mt-2babaef6-77b4b5bd9b-rzgmf\" (UID: \"8f89894d-a27e-44f9-8e7a-f1c909500cd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2babaef6-77b4b5bd9b-rzgmf" Nov 01 20:25:24.431632 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:24.431549 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f89894d-a27e-44f9-8e7a-f1c909500cd4-metrics-certs\") pod \"maas-api-e2e-mt-2babaef6-77b4b5bd9b-rzgmf\" (UID: \"8f89894d-a27e-44f9-8e7a-f1c909500cd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2babaef6-77b4b5bd9b-rzgmf" Nov 01 20:25:24.432193 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:24.432161 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f89894d-a27e-44f9-8e7a-f1c909500cd4-openshift-service-ca\") pod \"maas-api-e2e-mt-2babaef6-77b4b5bd9b-rzgmf\" (UID: \"8f89894d-a27e-44f9-8e7a-f1c909500cd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2babaef6-77b4b5bd9b-rzgmf" Nov 01 20:25:24.434302 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:24.434263 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f89894d-a27e-44f9-8e7a-f1c909500cd4-metrics-certs\") pod \"maas-api-e2e-mt-2babaef6-77b4b5bd9b-rzgmf\" (UID: \"8f89894d-a27e-44f9-8e7a-f1c909500cd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2babaef6-77b4b5bd9b-rzgmf" Nov 01 20:25:24.435253 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:24.435230 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f89894d-a27e-44f9-8e7a-f1c909500cd4-maas-api-tls\") pod \"maas-api-e2e-mt-2babaef6-77b4b5bd9b-rzgmf\" (UID: \"8f89894d-a27e-44f9-8e7a-f1c909500cd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2babaef6-77b4b5bd9b-rzgmf" Nov 01 20:25:24.441045 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:24.441019 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhsqg\" (UniqueName: \"kubernetes.io/projected/8f89894d-a27e-44f9-8e7a-f1c909500cd4-kube-api-access-xhsqg\") pod \"maas-api-e2e-mt-2babaef6-77b4b5bd9b-rzgmf\" (UID: \"8f89894d-a27e-44f9-8e7a-f1c909500cd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2babaef6-77b4b5bd9b-rzgmf" Nov 01 20:25:24.529111 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:24.529032 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2babaef6-77b4b5bd9b-rzgmf" Nov 01 20:25:24.820516 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:24.818430 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2babaef6-77b4b5bd9b-rzgmf"] Nov 01 20:25:24.825654 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:25:24.825608 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f89894d_a27e_44f9_8e7a_f1c909500cd4.slice/crio-6ae694c02b34f0455a4e80fd8f3cf34f7ab7c38ab49355ed1552131a1e70cadd WatchSource:0}: Error finding container 6ae694c02b34f0455a4e80fd8f3cf34f7ab7c38ab49355ed1552131a1e70cadd: Status 404 returned error can't find the container with id 6ae694c02b34f0455a4e80fd8f3cf34f7ab7c38ab49355ed1552131a1e70cadd Nov 01 20:25:25.646264 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:25.646182 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2babaef6-77b4b5bd9b-rzgmf" event={"ID":"8f89894d-a27e-44f9-8e7a-f1c909500cd4","Type":"ContainerStarted","Data":"8fb148b7ab9cd1e1e84aca6411e8baccc53f2471777bf0a01eb5679b450d38a9"} Nov 01 20:25:25.646264 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:25.646250 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2babaef6-77b4b5bd9b-rzgmf" event={"ID":"8f89894d-a27e-44f9-8e7a-f1c909500cd4","Type":"ContainerStarted","Data":"6ae694c02b34f0455a4e80fd8f3cf34f7ab7c38ab49355ed1552131a1e70cadd"} Nov 01 20:25:25.646728 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:25.646310 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2babaef6-77b4b5bd9b-rzgmf" Nov 01 20:25:25.665086 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:25.665036 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2babaef6-77b4b5bd9b-rzgmf" podStartSLOduration=1.195096193 podStartE2EDuration="1.665022288s" totalImagesPullingTime="469.926095ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:25:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:25:25.661512421 +0000 UTC m=+1571.434672946" watchObservedRunningTime="2026-11-01 20:25:25.665022288 +0000 UTC m=+1571.438182800" Nov 01 20:25:25.727400 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:25.727367 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-9ea5d2-66b589687d-jzxqb"] Nov 01 20:25:25.731608 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:25.731588 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9ea5d2-66b589687d-jzxqb" Nov 01 20:25:25.737809 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:25.737784 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-9ea5d2-serving-cert\"" Nov 01 20:25:25.744965 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:25.744942 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52998151-f859-4a20-b075-402dc0d0b6bc-maas-api-tls\") pod \"maas-api-e2e-extmod-b-9ea5d2-66b589687d-jzxqb\" (UID: \"52998151-f859-4a20-b075-402dc0d0b6bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9ea5d2-66b589687d-jzxqb" Nov 01 20:25:25.745080 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:25.744980 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52998151-f859-4a20-b075-402dc0d0b6bc-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-9ea5d2-66b589687d-jzxqb\" (UID: \"52998151-f859-4a20-b075-402dc0d0b6bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9ea5d2-66b589687d-jzxqb" Nov 01 20:25:25.745080 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:25.745010 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52998151-f859-4a20-b075-402dc0d0b6bc-metrics-certs\") pod \"maas-api-e2e-extmod-b-9ea5d2-66b589687d-jzxqb\" (UID: \"52998151-f859-4a20-b075-402dc0d0b6bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9ea5d2-66b589687d-jzxqb" Nov 01 20:25:25.745167 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:25.745137 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfcqc\" (UniqueName: \"kubernetes.io/projected/52998151-f859-4a20-b075-402dc0d0b6bc-kube-api-access-sfcqc\") pod \"maas-api-e2e-extmod-b-9ea5d2-66b589687d-jzxqb\" (UID: \"52998151-f859-4a20-b075-402dc0d0b6bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9ea5d2-66b589687d-jzxqb" Nov 01 20:25:25.750303 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:25.750262 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-9ea5d2-66b589687d-jzxqb"] Nov 01 20:25:25.846177 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:25.846146 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52998151-f859-4a20-b075-402dc0d0b6bc-maas-api-tls\") pod \"maas-api-e2e-extmod-b-9ea5d2-66b589687d-jzxqb\" (UID: \"52998151-f859-4a20-b075-402dc0d0b6bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9ea5d2-66b589687d-jzxqb" Nov 01 20:25:25.846321 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:25.846187 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52998151-f859-4a20-b075-402dc0d0b6bc-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-9ea5d2-66b589687d-jzxqb\" (UID: \"52998151-f859-4a20-b075-402dc0d0b6bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9ea5d2-66b589687d-jzxqb" Nov 01 20:25:25.846383 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:25.846339 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52998151-f859-4a20-b075-402dc0d0b6bc-metrics-certs\") pod \"maas-api-e2e-extmod-b-9ea5d2-66b589687d-jzxqb\" (UID: \"52998151-f859-4a20-b075-402dc0d0b6bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9ea5d2-66b589687d-jzxqb" Nov 01 20:25:25.846424 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:25.846404 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfcqc\" (UniqueName: \"kubernetes.io/projected/52998151-f859-4a20-b075-402dc0d0b6bc-kube-api-access-sfcqc\") pod \"maas-api-e2e-extmod-b-9ea5d2-66b589687d-jzxqb\" (UID: \"52998151-f859-4a20-b075-402dc0d0b6bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9ea5d2-66b589687d-jzxqb" Nov 01 20:25:25.846911 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:25.846889 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52998151-f859-4a20-b075-402dc0d0b6bc-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-9ea5d2-66b589687d-jzxqb\" (UID: \"52998151-f859-4a20-b075-402dc0d0b6bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9ea5d2-66b589687d-jzxqb" Nov 01 20:25:25.848647 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:25.848629 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52998151-f859-4a20-b075-402dc0d0b6bc-maas-api-tls\") pod \"maas-api-e2e-extmod-b-9ea5d2-66b589687d-jzxqb\" (UID: \"52998151-f859-4a20-b075-402dc0d0b6bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9ea5d2-66b589687d-jzxqb" Nov 01 20:25:25.848733 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:25.848691 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52998151-f859-4a20-b075-402dc0d0b6bc-metrics-certs\") pod \"maas-api-e2e-extmod-b-9ea5d2-66b589687d-jzxqb\" (UID: \"52998151-f859-4a20-b075-402dc0d0b6bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9ea5d2-66b589687d-jzxqb" Nov 01 20:25:25.858134 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:25.858111 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfcqc\" (UniqueName: \"kubernetes.io/projected/52998151-f859-4a20-b075-402dc0d0b6bc-kube-api-access-sfcqc\") pod \"maas-api-e2e-extmod-b-9ea5d2-66b589687d-jzxqb\" (UID: \"52998151-f859-4a20-b075-402dc0d0b6bc\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9ea5d2-66b589687d-jzxqb" Nov 01 20:25:26.047729 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.047692 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9ea5d2-66b589687d-jzxqb" Nov 01 20:25:26.324708 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.324680 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-9ea5d2-66b589687d-jzxqb"] Nov 01 20:25:26.325624 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:25:26.325599 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52998151_f859_4a20_b075_402dc0d0b6bc.slice/crio-38d3e7d120dc6554ed38d78c22d783251829ed78c9ac791ba3d0674f5a0baab2 WatchSource:0}: Error finding container 38d3e7d120dc6554ed38d78c22d783251829ed78c9ac791ba3d0674f5a0baab2: Status 404 returned error can't find the container with id 38d3e7d120dc6554ed38d78c22d783251829ed78c9ac791ba3d0674f5a0baab2 Nov 01 20:25:26.708226 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.707889 5935 generic.go:356] "Generic (PLEG): container finished" podID="86e63000-fc7e-4429-af13-4046198bca02" containerID="12e7c59c07a00198c5a91bebc1dadc8aeef05856ca87046334c3ed2b97dbab6d" exitCode=0 Nov 01 20:25:26.708226 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.707981 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6" event={"ID":"86e63000-fc7e-4429-af13-4046198bca02","Type":"ContainerDied","Data":"12e7c59c07a00198c5a91bebc1dadc8aeef05856ca87046334c3ed2b97dbab6d"} Nov 01 20:25:26.710680 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.710652 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9ea5d2-66b589687d-jzxqb" event={"ID":"52998151-f859-4a20-b075-402dc0d0b6bc","Type":"ContainerStarted","Data":"38d3e7d120dc6554ed38d78c22d783251829ed78c9ac791ba3d0674f5a0baab2"} Nov 01 20:25:26.712412 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.712394 5935 generic.go:356] "Generic (PLEG): container finished" podID="f8249130-8b25-468a-9904-2564d38fdb15" containerID="c74aa07d56563c2668fc0af4c188dabcff2d8a9a247f4dd9ec20736493faac6a" exitCode=0 Nov 01 20:25:26.712551 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.712525 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb" event={"ID":"f8249130-8b25-468a-9904-2564d38fdb15","Type":"ContainerDied","Data":"c74aa07d56563c2668fc0af4c188dabcff2d8a9a247f4dd9ec20736493faac6a"} Nov 01 20:25:26.712606 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.712589 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb" event={"ID":"f8249130-8b25-468a-9904-2564d38fdb15","Type":"ContainerDied","Data":"7e2423d2117f2f7abeed4a6c8529c6a1f47cdd52d6563aead0e6c5dc921b216c"} Nov 01 20:25:26.712640 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.712610 5935 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e2423d2117f2f7abeed4a6c8529c6a1f47cdd52d6563aead0e6c5dc921b216c" Nov 01 20:25:26.714955 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.714354 5935 generic.go:356] "Generic (PLEG): container finished" podID="16f53549-0a34-4825-a719-7ce0eff818ac" containerID="98a95c1f612cecb0a0ebe77340028fafeae5d7796fb3e419d9927fbc1491b62a" exitCode=0 Nov 01 20:25:26.715246 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.715178 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl" event={"ID":"16f53549-0a34-4825-a719-7ce0eff818ac","Type":"ContainerDied","Data":"98a95c1f612cecb0a0ebe77340028fafeae5d7796fb3e419d9927fbc1491b62a"} Nov 01 20:25:26.732240 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.727137 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb" Nov 01 20:25:26.732240 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.730874 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl" Nov 01 20:25:26.779717 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.779685 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8249130-8b25-468a-9904-2564d38fdb15-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8249130-8b25-468a-9904-2564d38fdb15-istio-data\") pod \"f8249130-8b25-468a-9904-2564d38fdb15\" (UID: \"f8249130-8b25-468a-9904-2564d38fdb15\") " Nov 01 20:25:26.779804 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.779740 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8249130-8b25-468a-9904-2564d38fdb15-istio-token\" (UniqueName: \"kubernetes.io/projected/f8249130-8b25-468a-9904-2564d38fdb15-istio-token\") pod \"f8249130-8b25-468a-9904-2564d38fdb15\" (UID: \"f8249130-8b25-468a-9904-2564d38fdb15\") " Nov 01 20:25:26.779804 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.779772 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8249130-8b25-468a-9904-2564d38fdb15-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8249130-8b25-468a-9904-2564d38fdb15-istio-envoy\") pod \"f8249130-8b25-468a-9904-2564d38fdb15\" (UID: \"f8249130-8b25-468a-9904-2564d38fdb15\") " Nov 01 20:25:26.779866 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.779822 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16f53549-0a34-4825-a719-7ce0eff818ac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16f53549-0a34-4825-a719-7ce0eff818ac-workload-socket\") pod \"16f53549-0a34-4825-a719-7ce0eff818ac\" (UID: \"16f53549-0a34-4825-a719-7ce0eff818ac\") " Nov 01 20:25:26.779909 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.779879 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8249130-8b25-468a-9904-2564d38fdb15-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8249130-8b25-468a-9904-2564d38fdb15-istiod-ca-cert\") pod \"f8249130-8b25-468a-9904-2564d38fdb15\" (UID: \"f8249130-8b25-468a-9904-2564d38fdb15\") " Nov 01 20:25:26.779956 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.779938 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16f53549-0a34-4825-a719-7ce0eff818ac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16f53549-0a34-4825-a719-7ce0eff818ac-istio-envoy\") pod \"16f53549-0a34-4825-a719-7ce0eff818ac\" (UID: \"16f53549-0a34-4825-a719-7ce0eff818ac\") " Nov 01 20:25:26.780009 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.779979 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f8249130-8b25-468a-9904-2564d38fdb15-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8249130-8b25-468a-9904-2564d38fdb15-istio-podinfo\") pod \"f8249130-8b25-468a-9904-2564d38fdb15\" (UID: \"f8249130-8b25-468a-9904-2564d38fdb15\") " Nov 01 20:25:26.780060 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.780045 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8249130-8b25-468a-9904-2564d38fdb15-kube-api-access-vltmj\" (UniqueName: \"kubernetes.io/projected/f8249130-8b25-468a-9904-2564d38fdb15-kube-api-access-vltmj\") pod \"f8249130-8b25-468a-9904-2564d38fdb15\" (UID: \"f8249130-8b25-468a-9904-2564d38fdb15\") " Nov 01 20:25:26.780120 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.780107 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16f53549-0a34-4825-a719-7ce0eff818ac-istio-token\" (UniqueName: \"kubernetes.io/projected/16f53549-0a34-4825-a719-7ce0eff818ac-istio-token\") pod \"16f53549-0a34-4825-a719-7ce0eff818ac\" (UID: \"16f53549-0a34-4825-a719-7ce0eff818ac\") " Nov 01 20:25:26.780172 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.780157 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8249130-8b25-468a-9904-2564d38fdb15-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8249130-8b25-468a-9904-2564d38fdb15-workload-socket\") pod \"f8249130-8b25-468a-9904-2564d38fdb15\" (UID: \"f8249130-8b25-468a-9904-2564d38fdb15\") " Nov 01 20:25:26.780237 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.780205 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16f53549-0a34-4825-a719-7ce0eff818ac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16f53549-0a34-4825-a719-7ce0eff818ac-istio-data\") pod \"16f53549-0a34-4825-a719-7ce0eff818ac\" (UID: \"16f53549-0a34-4825-a719-7ce0eff818ac\") " Nov 01 20:25:26.780358 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.780343 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/16f53549-0a34-4825-a719-7ce0eff818ac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16f53549-0a34-4825-a719-7ce0eff818ac-istio-podinfo\") pod \"16f53549-0a34-4825-a719-7ce0eff818ac\" (UID: \"16f53549-0a34-4825-a719-7ce0eff818ac\") " Nov 01 20:25:26.780523 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.780506 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16f53549-0a34-4825-a719-7ce0eff818ac-kube-api-access-bzszd\" (UniqueName: \"kubernetes.io/projected/16f53549-0a34-4825-a719-7ce0eff818ac-kube-api-access-bzszd\") pod \"16f53549-0a34-4825-a719-7ce0eff818ac\" (UID: \"16f53549-0a34-4825-a719-7ce0eff818ac\") " Nov 01 20:25:26.780590 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.780576 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8249130-8b25-468a-9904-2564d38fdb15-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8249130-8b25-468a-9904-2564d38fdb15-workload-certs\") pod \"f8249130-8b25-468a-9904-2564d38fdb15\" (UID: \"f8249130-8b25-468a-9904-2564d38fdb15\") " Nov 01 20:25:26.780708 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.780695 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16f53549-0a34-4825-a719-7ce0eff818ac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16f53549-0a34-4825-a719-7ce0eff818ac-credential-socket\") pod \"16f53549-0a34-4825-a719-7ce0eff818ac\" (UID: \"16f53549-0a34-4825-a719-7ce0eff818ac\") " Nov 01 20:25:26.780744 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.780726 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16f53549-0a34-4825-a719-7ce0eff818ac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16f53549-0a34-4825-a719-7ce0eff818ac-workload-certs\") pod \"16f53549-0a34-4825-a719-7ce0eff818ac\" (UID: \"16f53549-0a34-4825-a719-7ce0eff818ac\") " Nov 01 20:25:26.780786 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.780772 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8249130-8b25-468a-9904-2564d38fdb15-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8249130-8b25-468a-9904-2564d38fdb15-credential-socket\") pod \"f8249130-8b25-468a-9904-2564d38fdb15\" (UID: \"f8249130-8b25-468a-9904-2564d38fdb15\") " Nov 01 20:25:26.780823 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.780810 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16f53549-0a34-4825-a719-7ce0eff818ac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16f53549-0a34-4825-a719-7ce0eff818ac-istiod-ca-cert\") pod \"16f53549-0a34-4825-a719-7ce0eff818ac\" (UID: \"16f53549-0a34-4825-a719-7ce0eff818ac\") " Nov 01 20:25:26.782320 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.781555 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16f53549-0a34-4825-a719-7ce0eff818ac-istio-data" pod "16f53549-0a34-4825-a719-7ce0eff818ac" (UID: "16f53549-0a34-4825-a719-7ce0eff818ac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:25:26.782320 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.781769 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8249130-8b25-468a-9904-2564d38fdb15-istio-data" pod "f8249130-8b25-468a-9904-2564d38fdb15" (UID: "f8249130-8b25-468a-9904-2564d38fdb15"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:25:26.782474 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.782356 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16f53549-0a34-4825-a719-7ce0eff818ac-workload-socket" pod "16f53549-0a34-4825-a719-7ce0eff818ac" (UID: "16f53549-0a34-4825-a719-7ce0eff818ac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:25:26.783382 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.783039 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8249130-8b25-468a-9904-2564d38fdb15-credential-socket" pod "f8249130-8b25-468a-9904-2564d38fdb15" (UID: "f8249130-8b25-468a-9904-2564d38fdb15"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:25:26.785003 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.784090 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8249130-8b25-468a-9904-2564d38fdb15-istio-token" pod "f8249130-8b25-468a-9904-2564d38fdb15" (UID: "f8249130-8b25-468a-9904-2564d38fdb15"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:25:26.785453 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.785431 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16f53549-0a34-4825-a719-7ce0eff818ac-credential-socket" pod "16f53549-0a34-4825-a719-7ce0eff818ac" (UID: "16f53549-0a34-4825-a719-7ce0eff818ac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:25:26.785815 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.785787 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8249130-8b25-468a-9904-2564d38fdb15-workload-certs" pod "f8249130-8b25-468a-9904-2564d38fdb15" (UID: "f8249130-8b25-468a-9904-2564d38fdb15"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:25:26.786184 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.786164 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16f53549-0a34-4825-a719-7ce0eff818ac-workload-certs" pod "16f53549-0a34-4825-a719-7ce0eff818ac" (UID: "16f53549-0a34-4825-a719-7ce0eff818ac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:25:26.789311 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.786639 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8249130-8b25-468a-9904-2564d38fdb15-workload-socket" pod "f8249130-8b25-468a-9904-2564d38fdb15" (UID: "f8249130-8b25-468a-9904-2564d38fdb15"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:25:26.789311 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.786907 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16f53549-0a34-4825-a719-7ce0eff818ac-istio-token" pod "16f53549-0a34-4825-a719-7ce0eff818ac" (UID: "16f53549-0a34-4825-a719-7ce0eff818ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:25:26.789311 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.786909 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16f53549-0a34-4825-a719-7ce0eff818ac-istiod-ca-cert" pod "16f53549-0a34-4825-a719-7ce0eff818ac" (UID: "16f53549-0a34-4825-a719-7ce0eff818ac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:25:26.789311 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.787549 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8249130-8b25-468a-9904-2564d38fdb15-istiod-ca-cert" pod "f8249130-8b25-468a-9904-2564d38fdb15" (UID: "f8249130-8b25-468a-9904-2564d38fdb15"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:25:26.791426 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.791367 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8249130-8b25-468a-9904-2564d38fdb15-istio-podinfo" pod "f8249130-8b25-468a-9904-2564d38fdb15" (UID: "f8249130-8b25-468a-9904-2564d38fdb15"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:25:26.791720 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.791663 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8249130-8b25-468a-9904-2564d38fdb15-kube-api-access-vltmj" pod "f8249130-8b25-468a-9904-2564d38fdb15" (UID: "f8249130-8b25-468a-9904-2564d38fdb15"). InnerVolumeSpecName "kube-api-access-vltmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:25:26.791970 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.791939 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/16f53549-0a34-4825-a719-7ce0eff818ac-istio-podinfo" pod "16f53549-0a34-4825-a719-7ce0eff818ac" (UID: "16f53549-0a34-4825-a719-7ce0eff818ac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:25:26.791970 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.791951 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16f53549-0a34-4825-a719-7ce0eff818ac-kube-api-access-bzszd" pod "16f53549-0a34-4825-a719-7ce0eff818ac" (UID: "16f53549-0a34-4825-a719-7ce0eff818ac"). InnerVolumeSpecName "kube-api-access-bzszd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:25:26.793993 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.793973 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16f53549-0a34-4825-a719-7ce0eff818ac-istio-envoy" pod "16f53549-0a34-4825-a719-7ce0eff818ac" (UID: "16f53549-0a34-4825-a719-7ce0eff818ac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:25:26.800941 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.800652 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6" Nov 01 20:25:26.800941 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.800844 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8249130-8b25-468a-9904-2564d38fdb15-istio-envoy" pod "f8249130-8b25-468a-9904-2564d38fdb15" (UID: "f8249130-8b25-468a-9904-2564d38fdb15"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:25:26.882871 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.882802 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86e63000-fc7e-4429-af13-4046198bca02-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86e63000-fc7e-4429-af13-4046198bca02-istiod-ca-cert\") pod \"86e63000-fc7e-4429-af13-4046198bca02\" (UID: \"86e63000-fc7e-4429-af13-4046198bca02\") " Nov 01 20:25:26.883012 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.882918 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86e63000-fc7e-4429-af13-4046198bca02-kube-api-access-67czq\" (UniqueName: \"kubernetes.io/projected/86e63000-fc7e-4429-af13-4046198bca02-kube-api-access-67czq\") pod \"86e63000-fc7e-4429-af13-4046198bca02\" (UID: \"86e63000-fc7e-4429-af13-4046198bca02\") " Nov 01 20:25:26.883012 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.882950 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/86e63000-fc7e-4429-af13-4046198bca02-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86e63000-fc7e-4429-af13-4046198bca02-istio-podinfo\") pod \"86e63000-fc7e-4429-af13-4046198bca02\" (UID: \"86e63000-fc7e-4429-af13-4046198bca02\") " Nov 01 20:25:26.883012 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.883005 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86e63000-fc7e-4429-af13-4046198bca02-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86e63000-fc7e-4429-af13-4046198bca02-credential-socket\") pod \"86e63000-fc7e-4429-af13-4046198bca02\" (UID: \"86e63000-fc7e-4429-af13-4046198bca02\") " Nov 01 20:25:26.883153 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.883049 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86e63000-fc7e-4429-af13-4046198bca02-istio-token\" (UniqueName: \"kubernetes.io/projected/86e63000-fc7e-4429-af13-4046198bca02-istio-token\") pod \"86e63000-fc7e-4429-af13-4046198bca02\" (UID: \"86e63000-fc7e-4429-af13-4046198bca02\") " Nov 01 20:25:26.883153 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.883085 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86e63000-fc7e-4429-af13-4046198bca02-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86e63000-fc7e-4429-af13-4046198bca02-istio-envoy\") pod \"86e63000-fc7e-4429-af13-4046198bca02\" (UID: \"86e63000-fc7e-4429-af13-4046198bca02\") " Nov 01 20:25:26.883221 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.883156 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86e63000-fc7e-4429-af13-4046198bca02-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86e63000-fc7e-4429-af13-4046198bca02-workload-certs\") pod \"86e63000-fc7e-4429-af13-4046198bca02\" (UID: \"86e63000-fc7e-4429-af13-4046198bca02\") " Nov 01 20:25:26.883221 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.883202 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86e63000-fc7e-4429-af13-4046198bca02-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86e63000-fc7e-4429-af13-4046198bca02-istio-data\") pod \"86e63000-fc7e-4429-af13-4046198bca02\" (UID: \"86e63000-fc7e-4429-af13-4046198bca02\") " Nov 01 20:25:26.883315 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.883253 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86e63000-fc7e-4429-af13-4046198bca02-istiod-ca-cert" pod "86e63000-fc7e-4429-af13-4046198bca02" (UID: "86e63000-fc7e-4429-af13-4046198bca02"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:25:26.883315 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.883277 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86e63000-fc7e-4429-af13-4046198bca02-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86e63000-fc7e-4429-af13-4046198bca02-workload-socket\") pod \"86e63000-fc7e-4429-af13-4046198bca02\" (UID: \"86e63000-fc7e-4429-af13-4046198bca02\") " Nov 01 20:25:26.883781 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.883757 5935 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16f53549-0a34-4825-a719-7ce0eff818ac-istio-podinfo\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:25:26.883964 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.883783 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzszd\" (UniqueName: \"kubernetes.io/projected/16f53549-0a34-4825-a719-7ce0eff818ac-kube-api-access-bzszd\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:25:26.883964 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.883799 5935 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8249130-8b25-468a-9904-2564d38fdb15-workload-certs\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:25:26.883964 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.883812 5935 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16f53549-0a34-4825-a719-7ce0eff818ac-credential-socket\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:25:26.883964 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.883824 5935 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16f53549-0a34-4825-a719-7ce0eff818ac-workload-certs\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:25:26.883964 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.883837 5935 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8249130-8b25-468a-9904-2564d38fdb15-credential-socket\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:25:26.883964 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.883849 5935 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16f53549-0a34-4825-a719-7ce0eff818ac-istiod-ca-cert\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:25:26.883964 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.883862 5935 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8249130-8b25-468a-9904-2564d38fdb15-istio-data\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:25:26.883964 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.883876 5935 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8249130-8b25-468a-9904-2564d38fdb15-istio-token\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:25:26.883964 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.883889 5935 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8249130-8b25-468a-9904-2564d38fdb15-istio-envoy\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:25:26.883964 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.883902 5935 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16f53549-0a34-4825-a719-7ce0eff818ac-workload-socket\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:25:26.883964 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.883916 5935 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86e63000-fc7e-4429-af13-4046198bca02-istiod-ca-cert\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:25:26.883964 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.883931 5935 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8249130-8b25-468a-9904-2564d38fdb15-istiod-ca-cert\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:25:26.883964 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.883943 5935 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16f53549-0a34-4825-a719-7ce0eff818ac-istio-envoy\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:25:26.883964 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.883956 5935 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8249130-8b25-468a-9904-2564d38fdb15-istio-podinfo\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:25:26.883964 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.883968 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vltmj\" (UniqueName: \"kubernetes.io/projected/f8249130-8b25-468a-9904-2564d38fdb15-kube-api-access-vltmj\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:25:26.884882 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.883982 5935 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16f53549-0a34-4825-a719-7ce0eff818ac-istio-token\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:25:26.884882 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.883994 5935 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8249130-8b25-468a-9904-2564d38fdb15-workload-socket\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:25:26.884882 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.884008 5935 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16f53549-0a34-4825-a719-7ce0eff818ac-istio-data\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:25:26.884882 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.884221 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86e63000-fc7e-4429-af13-4046198bca02-workload-socket" pod "86e63000-fc7e-4429-af13-4046198bca02" (UID: "86e63000-fc7e-4429-af13-4046198bca02"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:25:26.885120 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.885099 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86e63000-fc7e-4429-af13-4046198bca02-istio-data" pod "86e63000-fc7e-4429-af13-4046198bca02" (UID: "86e63000-fc7e-4429-af13-4046198bca02"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:25:26.885178 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.885162 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86e63000-fc7e-4429-af13-4046198bca02-workload-certs" pod "86e63000-fc7e-4429-af13-4046198bca02" (UID: "86e63000-fc7e-4429-af13-4046198bca02"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:25:26.885394 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.885376 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86e63000-fc7e-4429-af13-4046198bca02-credential-socket" pod "86e63000-fc7e-4429-af13-4046198bca02" (UID: "86e63000-fc7e-4429-af13-4046198bca02"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:25:26.885893 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.885863 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86e63000-fc7e-4429-af13-4046198bca02-istio-token" pod "86e63000-fc7e-4429-af13-4046198bca02" (UID: "86e63000-fc7e-4429-af13-4046198bca02"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:25:26.886453 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.886279 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86e63000-fc7e-4429-af13-4046198bca02-istio-podinfo" pod "86e63000-fc7e-4429-af13-4046198bca02" (UID: "86e63000-fc7e-4429-af13-4046198bca02"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:25:26.886728 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.886702 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86e63000-fc7e-4429-af13-4046198bca02-istio-envoy" pod "86e63000-fc7e-4429-af13-4046198bca02" (UID: "86e63000-fc7e-4429-af13-4046198bca02"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:25:26.887356 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.887334 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86e63000-fc7e-4429-af13-4046198bca02-kube-api-access-67czq" pod "86e63000-fc7e-4429-af13-4046198bca02" (UID: "86e63000-fc7e-4429-af13-4046198bca02"). InnerVolumeSpecName "kube-api-access-67czq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:25:26.984994 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.984966 5935 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86e63000-fc7e-4429-af13-4046198bca02-workload-certs\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:25:26.984994 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.984990 5935 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86e63000-fc7e-4429-af13-4046198bca02-istio-data\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:25:26.984994 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.984999 5935 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86e63000-fc7e-4429-af13-4046198bca02-workload-socket\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:25:26.985190 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.985011 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67czq\" (UniqueName: \"kubernetes.io/projected/86e63000-fc7e-4429-af13-4046198bca02-kube-api-access-67czq\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:25:26.985190 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.985020 5935 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86e63000-fc7e-4429-af13-4046198bca02-istio-podinfo\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:25:26.985190 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.985032 5935 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86e63000-fc7e-4429-af13-4046198bca02-credential-socket\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:25:26.985190 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.985041 5935 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86e63000-fc7e-4429-af13-4046198bca02-istio-token\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:25:26.985190 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:26.985050 5935 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86e63000-fc7e-4429-af13-4046198bca02-istio-envoy\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:25:27.576495 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:27.576469 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9ea5d2-78b67f847f-j8dj9" Nov 01 20:25:27.576653 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:27.576529 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-9ea5d2-6c96fd4f45-lxrs4" Nov 01 20:25:27.728469 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:27.728389 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl" Nov 01 20:25:27.728469 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:27.728409 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl" event={"ID":"16f53549-0a34-4825-a719-7ce0eff818ac","Type":"ContainerDied","Data":"ad713a49c3e0f501d1da7a631abbff41587353865710b2f576fab6298ca3b282"} Nov 01 20:25:27.728469 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:27.728468 5935 scope.go:122] "RemoveContainer" containerID="98a95c1f612cecb0a0ebe77340028fafeae5d7796fb3e419d9927fbc1491b62a" Nov 01 20:25:27.730038 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:27.730015 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6" event={"ID":"86e63000-fc7e-4429-af13-4046198bca02","Type":"ContainerDied","Data":"7c96c1917d1455f7d7b556adae2e0a2a71b804af0a8cdb3350d54dce29ea2a1f"} Nov 01 20:25:27.730128 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:27.730046 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6" Nov 01 20:25:27.732068 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:27.732030 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9ea5d2-66b589687d-jzxqb" event={"ID":"52998151-f859-4a20-b075-402dc0d0b6bc","Type":"ContainerStarted","Data":"bcc92c5f5755033ce81379889ee4e5f8c7c012600452657fbffa21ef372b04b2"} Nov 01 20:25:27.732203 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:27.732071 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb" Nov 01 20:25:27.732413 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:27.732394 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9ea5d2-66b589687d-jzxqb" Nov 01 20:25:27.739270 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:27.739255 5935 scope.go:122] "RemoveContainer" containerID="12e7c59c07a00198c5a91bebc1dadc8aeef05856ca87046334c3ed2b97dbab6d" Nov 01 20:25:27.759061 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:27.759023 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9ea5d2-66b589687d-jzxqb" podStartSLOduration=2.349550681 podStartE2EDuration="2.759011883s" totalImagesPullingTime="409.461202ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:25:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:25:27.756102444 +0000 UTC m=+1573.529262986" watchObservedRunningTime="2026-11-01 20:25:27.759011883 +0000 UTC m=+1573.532172395" Nov 01 20:25:27.771430 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:27.771406 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl"] Nov 01 20:25:27.777528 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:27.777504 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-pl7jl"] Nov 01 20:25:27.787781 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:27.787759 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb"] Nov 01 20:25:27.791794 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:27.791773 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-ts8nb"] Nov 01 20:25:27.801981 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:27.801961 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6"] Nov 01 20:25:27.808669 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:27.808649 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e27ae7-openshift-default-74cd56c8bc-n9gw6"] Nov 01 20:25:28.354202 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:28.354172 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16f53549-0a34-4825-a719-7ce0eff818ac" path="/var/lib/kubelet/pods/16f53549-0a34-4825-a719-7ce0eff818ac/volumes" Nov 01 20:25:28.354669 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:28.354654 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86e63000-fc7e-4429-af13-4046198bca02" path="/var/lib/kubelet/pods/86e63000-fc7e-4429-af13-4046198bca02/volumes" Nov 01 20:25:28.355072 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:28.355056 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8249130-8b25-468a-9904-2564d38fdb15" path="/var/lib/kubelet/pods/f8249130-8b25-468a-9904-2564d38fdb15/volumes" Nov 01 20:25:31.720570 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:31.720541 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2babaef6-77b4b5bd9b-rzgmf" Nov 01 20:25:33.746644 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:33.746615 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9ea5d2-66b589687d-jzxqb" Nov 01 20:25:35.844659 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:25:35.844627 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:25:44.572474 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:44.572431 5935 patch_prober.go:29] interesting pod/e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 20:25:44.572935 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:44.572513 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p" podUID="0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:25:47.930915 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:47.930884 5935 generic.go:356] "Generic (PLEG): container finished" podID="0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0" containerID="9e0b2bf5d91d0d6af1a2f8176065780be9ccb24b68e00573db4a0ffb1c2c086a" exitCode=0 Nov 01 20:25:47.931332 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:47.930955 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p" event={"ID":"0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0","Type":"ContainerDied","Data":"9e0b2bf5d91d0d6af1a2f8176065780be9ccb24b68e00573db4a0ffb1c2c086a"} Nov 01 20:25:47.931391 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:47.931376 5935 scope.go:122] "RemoveContainer" containerID="9e0b2bf5d91d0d6af1a2f8176065780be9ccb24b68e00573db4a0ffb1c2c086a" Nov 01 20:25:47.932944 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:47.932915 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:25:47.933022 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:47.932983 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:25:47.933059 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:47.933047 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:25:48.272760 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:48.272719 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-2babaef6/e2e-lifecycle-model-2babaef6-kserve-c7444bf-twlvv"] Nov 01 20:25:48.970587 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:48.970550 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p" event={"ID":"0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0","Type":"ContainerStarted","Data":"2e67ceba65fcea7b37dd46674360ec26ea655750622f4ebc21bbac674303a3bf"} Nov 01 20:25:48.981756 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:48.981729 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p" Nov 01 20:25:49.981834 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:49.981791 5935 patch_prober.go:29] interesting pod/e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 01 20:25:49.982247 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:49.981850 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p" podUID="0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 01 20:25:50.981601 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:50.981568 5935 patch_prober.go:29] interesting pod/e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 01 20:25:50.981791 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:50.981634 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p" podUID="0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 01 20:25:51.981863 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:51.981830 5935 patch_prober.go:29] interesting pod/e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 01 20:25:51.982274 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:51.981885 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p" podUID="0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 01 20:25:52.981569 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:52.981540 5935 patch_prober.go:29] interesting pod/e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 01 20:25:52.981736 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:52.981600 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p" podUID="0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 01 20:25:53.982131 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:53.982091 5935 patch_prober.go:29] interesting pod/e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Nov 01 20:25:53.982574 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:53.982144 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p" podUID="0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Nov 01 20:25:54.986458 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:54.986430 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p" Nov 01 20:25:54.986926 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:54.986704 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p" Nov 01 20:25:54.987405 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:25:54.987389 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p" Nov 01 20:26:03.301460 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:03.301420 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-2babaef6/e2e-lifecycle-model-2babaef6-kserve-c7444bf-twlvv" podUID="098cc87f-f041-4d32-b909-be5a16ced4d8" containerName="main" containerID="cri-o://c3219d8fa524b7a216f1d9b201635e74a23ee0ff42917441074b35874257e44c" gracePeriod=45 Nov 01 20:26:03.610375 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:03.610344 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2babaef6/e2e-lifecycle-model-2babaef6-kserve-c7444bf-twlvv" Nov 01 20:26:03.776576 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:03.776539 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/098cc87f-f041-4d32-b909-be5a16ced4d8-tls-certs\" (UniqueName: \"kubernetes.io/secret/098cc87f-f041-4d32-b909-be5a16ced4d8-tls-certs\") pod \"098cc87f-f041-4d32-b909-be5a16ced4d8\" (UID: \"098cc87f-f041-4d32-b909-be5a16ced4d8\") " Nov 01 20:26:03.776812 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:03.776623 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/098cc87f-f041-4d32-b909-be5a16ced4d8-home\" (UniqueName: \"kubernetes.io/empty-dir/098cc87f-f041-4d32-b909-be5a16ced4d8-home\") pod \"098cc87f-f041-4d32-b909-be5a16ced4d8\" (UID: \"098cc87f-f041-4d32-b909-be5a16ced4d8\") " Nov 01 20:26:03.776812 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:03.776695 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/098cc87f-f041-4d32-b909-be5a16ced4d8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/098cc87f-f041-4d32-b909-be5a16ced4d8-dshm\") pod \"098cc87f-f041-4d32-b909-be5a16ced4d8\" (UID: \"098cc87f-f041-4d32-b909-be5a16ced4d8\") " Nov 01 20:26:03.776812 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:03.776787 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/098cc87f-f041-4d32-b909-be5a16ced4d8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/098cc87f-f041-4d32-b909-be5a16ced4d8-model-cache\") pod \"098cc87f-f041-4d32-b909-be5a16ced4d8\" (UID: \"098cc87f-f041-4d32-b909-be5a16ced4d8\") " Nov 01 20:26:03.776986 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:03.776914 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/098cc87f-f041-4d32-b909-be5a16ced4d8-home" pod "098cc87f-f041-4d32-b909-be5a16ced4d8" (UID: "098cc87f-f041-4d32-b909-be5a16ced4d8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:26:03.776986 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:03.776929 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/098cc87f-f041-4d32-b909-be5a16ced4d8-kube-api-access-pbp8n\" (UniqueName: \"kubernetes.io/projected/098cc87f-f041-4d32-b909-be5a16ced4d8-kube-api-access-pbp8n\") pod \"098cc87f-f041-4d32-b909-be5a16ced4d8\" (UID: \"098cc87f-f041-4d32-b909-be5a16ced4d8\") " Nov 01 20:26:03.776986 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:03.776974 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/098cc87f-f041-4d32-b909-be5a16ced4d8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/098cc87f-f041-4d32-b909-be5a16ced4d8-tmp-dir\") pod \"098cc87f-f041-4d32-b909-be5a16ced4d8\" (UID: \"098cc87f-f041-4d32-b909-be5a16ced4d8\") " Nov 01 20:26:03.777131 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:03.777024 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/098cc87f-f041-4d32-b909-be5a16ced4d8-model-cache" pod "098cc87f-f041-4d32-b909-be5a16ced4d8" (UID: "098cc87f-f041-4d32-b909-be5a16ced4d8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:26:03.777328 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:03.777269 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/098cc87f-f041-4d32-b909-be5a16ced4d8-tmp-dir" pod "098cc87f-f041-4d32-b909-be5a16ced4d8" (UID: "098cc87f-f041-4d32-b909-be5a16ced4d8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:26:03.777483 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:03.777416 5935 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/098cc87f-f041-4d32-b909-be5a16ced4d8-model-cache\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:26:03.777483 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:03.777447 5935 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/098cc87f-f041-4d32-b909-be5a16ced4d8-tmp-dir\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:26:03.777483 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:03.777456 5935 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/098cc87f-f041-4d32-b909-be5a16ced4d8-home\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:26:03.778983 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:03.778961 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/098cc87f-f041-4d32-b909-be5a16ced4d8-dshm" pod "098cc87f-f041-4d32-b909-be5a16ced4d8" (UID: "098cc87f-f041-4d32-b909-be5a16ced4d8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:26:03.779079 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:03.779010 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/098cc87f-f041-4d32-b909-be5a16ced4d8-kube-api-access-pbp8n" pod "098cc87f-f041-4d32-b909-be5a16ced4d8" (UID: "098cc87f-f041-4d32-b909-be5a16ced4d8"). InnerVolumeSpecName "kube-api-access-pbp8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:26:03.779079 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:03.779069 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/098cc87f-f041-4d32-b909-be5a16ced4d8-tls-certs" pod "098cc87f-f041-4d32-b909-be5a16ced4d8" (UID: "098cc87f-f041-4d32-b909-be5a16ced4d8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:26:03.878251 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:03.878192 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbp8n\" (UniqueName: \"kubernetes.io/projected/098cc87f-f041-4d32-b909-be5a16ced4d8-kube-api-access-pbp8n\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:26:03.878251 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:03.878217 5935 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/098cc87f-f041-4d32-b909-be5a16ced4d8-tls-certs\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:26:03.878251 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:03.878227 5935 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/098cc87f-f041-4d32-b909-be5a16ced4d8-dshm\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:26:04.083112 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:04.083074 5935 generic.go:356] "Generic (PLEG): container finished" podID="098cc87f-f041-4d32-b909-be5a16ced4d8" containerID="c3219d8fa524b7a216f1d9b201635e74a23ee0ff42917441074b35874257e44c" exitCode=0 Nov 01 20:26:04.083323 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:04.083136 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2babaef6/e2e-lifecycle-model-2babaef6-kserve-c7444bf-twlvv" Nov 01 20:26:04.083323 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:04.083164 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2babaef6/e2e-lifecycle-model-2babaef6-kserve-c7444bf-twlvv" event={"ID":"098cc87f-f041-4d32-b909-be5a16ced4d8","Type":"ContainerDied","Data":"c3219d8fa524b7a216f1d9b201635e74a23ee0ff42917441074b35874257e44c"} Nov 01 20:26:04.083323 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:04.083221 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2babaef6/e2e-lifecycle-model-2babaef6-kserve-c7444bf-twlvv" event={"ID":"098cc87f-f041-4d32-b909-be5a16ced4d8","Type":"ContainerDied","Data":"0482572908e62aca02d2cbca0dd7343b4ece63c077d5cf3295d6094d25cc4d9b"} Nov 01 20:26:04.083323 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:04.083244 5935 scope.go:122] "RemoveContainer" containerID="c3219d8fa524b7a216f1d9b201635e74a23ee0ff42917441074b35874257e44c" Nov 01 20:26:04.094072 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:04.094054 5935 scope.go:122] "RemoveContainer" containerID="c3219d8fa524b7a216f1d9b201635e74a23ee0ff42917441074b35874257e44c" Nov 01 20:26:04.094384 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:26:04.094364 5935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3219d8fa524b7a216f1d9b201635e74a23ee0ff42917441074b35874257e44c\": container with ID starting with c3219d8fa524b7a216f1d9b201635e74a23ee0ff42917441074b35874257e44c not found: ID does not exist" containerID="c3219d8fa524b7a216f1d9b201635e74a23ee0ff42917441074b35874257e44c" Nov 01 20:26:04.094440 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:04.094392 5935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3219d8fa524b7a216f1d9b201635e74a23ee0ff42917441074b35874257e44c"} err="failed to get container status \"c3219d8fa524b7a216f1d9b201635e74a23ee0ff42917441074b35874257e44c\": rpc error: code = NotFound desc = could not find container \"c3219d8fa524b7a216f1d9b201635e74a23ee0ff42917441074b35874257e44c\": container with ID starting with c3219d8fa524b7a216f1d9b201635e74a23ee0ff42917441074b35874257e44c not found: ID does not exist" Nov 01 20:26:04.111357 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:04.111330 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-2babaef6/e2e-lifecycle-model-2babaef6-kserve-c7444bf-twlvv"] Nov 01 20:26:04.116333 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:04.116311 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-2babaef6/e2e-lifecycle-model-2babaef6-kserve-c7444bf-twlvv"] Nov 01 20:26:04.353827 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:04.353785 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="098cc87f-f041-4d32-b909-be5a16ced4d8" path="/var/lib/kubelet/pods/098cc87f-f041-4d32-b909-be5a16ced4d8/volumes" Nov 01 20:26:04.539940 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:04.539905 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-2babaef6/e2e-lifecycle-model-2babaef6-kserve-c7444bf-twlvv" podUID="098cc87f-f041-4d32-b909-be5a16ced4d8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 01 20:26:05.843955 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:26:05.843925 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:26:12.559809 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:12.559771 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s"] Nov 01 20:26:12.560312 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:12.560043 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s" podUID="b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae" containerName="istio-proxy" containerID="cri-o://9d37d4afbd227de761ef4dde23cedcf9d9a46527ba784e027d6d8d49548bf6c4" gracePeriod=30 Nov 01 20:26:14.837423 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:14.837098 5935 scope.go:122] "RemoveContainer" containerID="c74aa07d56563c2668fc0af4c188dabcff2d8a9a247f4dd9ec20736493faac6a" Nov 01 20:26:17.818148 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:17.818124 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s" Nov 01 20:26:17.943351 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:17.943320 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae-istio-token\" (UniqueName: \"kubernetes.io/projected/b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae-istio-token\") pod \"b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae\" (UID: \"b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae\") " Nov 01 20:26:17.943522 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:17.943371 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae-istiod-ca-cert\") pod \"b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae\" (UID: \"b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae\") " Nov 01 20:26:17.943522 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:17.943403 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae-istio-podinfo\") pod \"b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae\" (UID: \"b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae\") " Nov 01 20:26:17.943522 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:17.943446 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae-istio-envoy\") pod \"b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae\" (UID: \"b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae\") " Nov 01 20:26:17.943522 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:17.943513 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae-workload-certs\") pod \"b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae\" (UID: \"b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae\") " Nov 01 20:26:17.943754 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:17.943607 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae-credential-socket\") pod \"b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae\" (UID: \"b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae\") " Nov 01 20:26:17.943754 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:17.943670 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae-istio-data\") pod \"b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae\" (UID: \"b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae\") " Nov 01 20:26:17.943754 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:17.943712 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae-kube-api-access-2s6jg\" (UniqueName: \"kubernetes.io/projected/b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae-kube-api-access-2s6jg\") pod \"b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae\" (UID: \"b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae\") " Nov 01 20:26:17.943914 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:17.943844 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae-workload-socket\") pod \"b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae\" (UID: \"b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae\") " Nov 01 20:26:17.943914 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:17.943876 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae-workload-certs" pod "b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae" (UID: "b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:26:17.943914 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:17.943892 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae-credential-socket" pod "b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae" (UID: "b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:26:17.944089 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:17.943920 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae-istiod-ca-cert" pod "b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae" (UID: "b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:26:17.944089 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:17.943984 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae-istio-data" pod "b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae" (UID: "b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:26:17.944221 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:17.944190 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae-workload-socket" pod "b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae" (UID: "b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:26:17.944402 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:17.944382 5935 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae-workload-socket\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:26:17.944497 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:17.944404 5935 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae-istiod-ca-cert\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:26:17.944497 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:17.944414 5935 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae-workload-certs\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:26:17.944497 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:17.944423 5935 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae-credential-socket\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:26:17.944497 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:17.944436 5935 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae-istio-data\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:26:17.945646 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:17.945620 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae-istio-envoy" pod "b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae" (UID: "b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:26:17.945778 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:17.945757 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae-istio-token" pod "b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae" (UID: "b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:26:17.945883 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:17.945866 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae-istio-podinfo" pod "b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae" (UID: "b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:26:17.946108 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:17.946088 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae-kube-api-access-2s6jg" pod "b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae" (UID: "b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae"). InnerVolumeSpecName "kube-api-access-2s6jg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:26:18.045821 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:18.045781 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2s6jg\" (UniqueName: \"kubernetes.io/projected/b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae-kube-api-access-2s6jg\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:26:18.045821 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:18.045817 5935 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae-istio-token\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:26:18.045821 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:18.045828 5935 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae-istio-podinfo\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:26:18.046056 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:18.045838 5935 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae-istio-envoy\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:26:18.178004 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:18.177974 5935 generic.go:356] "Generic (PLEG): container finished" podID="b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae" containerID="9d37d4afbd227de761ef4dde23cedcf9d9a46527ba784e027d6d8d49548bf6c4" exitCode=0 Nov 01 20:26:18.178172 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:18.178024 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s" Nov 01 20:26:18.178172 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:18.178038 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s" event={"ID":"b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae","Type":"ContainerDied","Data":"9d37d4afbd227de761ef4dde23cedcf9d9a46527ba784e027d6d8d49548bf6c4"} Nov 01 20:26:18.178172 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:18.178096 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s" event={"ID":"b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae","Type":"ContainerDied","Data":"e9842ed11234cef35e48d2de47a3c66cd884efb203330d0ea0caf1cc08742fb4"} Nov 01 20:26:18.178172 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:18.178118 5935 scope.go:122] "RemoveContainer" containerID="9d37d4afbd227de761ef4dde23cedcf9d9a46527ba784e027d6d8d49548bf6c4" Nov 01 20:26:18.188630 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:18.188613 5935 scope.go:122] "RemoveContainer" containerID="9d37d4afbd227de761ef4dde23cedcf9d9a46527ba784e027d6d8d49548bf6c4" Nov 01 20:26:18.188889 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:26:18.188875 5935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d37d4afbd227de761ef4dde23cedcf9d9a46527ba784e027d6d8d49548bf6c4\": container with ID starting with 9d37d4afbd227de761ef4dde23cedcf9d9a46527ba784e027d6d8d49548bf6c4 not found: ID does not exist" containerID="9d37d4afbd227de761ef4dde23cedcf9d9a46527ba784e027d6d8d49548bf6c4" Nov 01 20:26:18.188928 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:18.188896 5935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d37d4afbd227de761ef4dde23cedcf9d9a46527ba784e027d6d8d49548bf6c4"} err="failed to get container status \"9d37d4afbd227de761ef4dde23cedcf9d9a46527ba784e027d6d8d49548bf6c4\": rpc error: code = NotFound desc = could not find container \"9d37d4afbd227de761ef4dde23cedcf9d9a46527ba784e027d6d8d49548bf6c4\": container with ID starting with 9d37d4afbd227de761ef4dde23cedcf9d9a46527ba784e027d6d8d49548bf6c4 not found: ID does not exist" Nov 01 20:26:18.201441 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:18.201388 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s"] Nov 01 20:26:18.206425 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:18.206398 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-vsq9s"] Nov 01 20:26:18.354365 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:18.354336 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae" path="/var/lib/kubelet/pods/b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae/volumes" Nov 01 20:26:31.656144 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:31.656049 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2babaef6-77b4b5bd9b-rzgmf"] Nov 01 20:26:31.656707 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:31.656303 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2babaef6-77b4b5bd9b-rzgmf" podUID="8f89894d-a27e-44f9-8e7a-f1c909500cd4" containerName="maas-api" containerID="cri-o://8fb148b7ab9cd1e1e84aca6411e8baccc53f2471777bf0a01eb5679b450d38a9" gracePeriod=30 Nov 01 20:26:31.715921 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:31.715877 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2babaef6-77b4b5bd9b-rzgmf" podUID="8f89894d-a27e-44f9-8e7a-f1c909500cd4" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.108:8443/health\": dial tcp 10.133.0.108:8443: connect: connection refused" Nov 01 20:26:31.989856 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:31.989830 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2babaef6-77b4b5bd9b-rzgmf" Nov 01 20:26:32.145121 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:32.145090 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f89894d-a27e-44f9-8e7a-f1c909500cd4-kube-api-access-xhsqg\" (UniqueName: \"kubernetes.io/projected/8f89894d-a27e-44f9-8e7a-f1c909500cd4-kube-api-access-xhsqg\") pod \"8f89894d-a27e-44f9-8e7a-f1c909500cd4\" (UID: \"8f89894d-a27e-44f9-8e7a-f1c909500cd4\") " Nov 01 20:26:32.145542 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:32.145252 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f89894d-a27e-44f9-8e7a-f1c909500cd4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f89894d-a27e-44f9-8e7a-f1c909500cd4-maas-api-tls\") pod \"8f89894d-a27e-44f9-8e7a-f1c909500cd4\" (UID: \"8f89894d-a27e-44f9-8e7a-f1c909500cd4\") " Nov 01 20:26:32.145542 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:32.145465 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f89894d-a27e-44f9-8e7a-f1c909500cd4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f89894d-a27e-44f9-8e7a-f1c909500cd4-openshift-service-ca\") pod \"8f89894d-a27e-44f9-8e7a-f1c909500cd4\" (UID: \"8f89894d-a27e-44f9-8e7a-f1c909500cd4\") " Nov 01 20:26:32.146051 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:32.145540 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f89894d-a27e-44f9-8e7a-f1c909500cd4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f89894d-a27e-44f9-8e7a-f1c909500cd4-metrics-certs\") pod \"8f89894d-a27e-44f9-8e7a-f1c909500cd4\" (UID: \"8f89894d-a27e-44f9-8e7a-f1c909500cd4\") " Nov 01 20:26:32.147556 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:32.147533 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f89894d-a27e-44f9-8e7a-f1c909500cd4-openshift-service-ca" pod "8f89894d-a27e-44f9-8e7a-f1c909500cd4" (UID: "8f89894d-a27e-44f9-8e7a-f1c909500cd4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:26:32.151019 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:32.150913 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f89894d-a27e-44f9-8e7a-f1c909500cd4-kube-api-access-xhsqg" pod "8f89894d-a27e-44f9-8e7a-f1c909500cd4" (UID: "8f89894d-a27e-44f9-8e7a-f1c909500cd4"). InnerVolumeSpecName "kube-api-access-xhsqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:26:32.151019 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:32.150996 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f89894d-a27e-44f9-8e7a-f1c909500cd4-maas-api-tls" pod "8f89894d-a27e-44f9-8e7a-f1c909500cd4" (UID: "8f89894d-a27e-44f9-8e7a-f1c909500cd4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:26:32.153066 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:32.153038 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f89894d-a27e-44f9-8e7a-f1c909500cd4-metrics-certs" pod "8f89894d-a27e-44f9-8e7a-f1c909500cd4" (UID: "8f89894d-a27e-44f9-8e7a-f1c909500cd4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:26:32.247717 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:32.247641 5935 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f89894d-a27e-44f9-8e7a-f1c909500cd4-openshift-service-ca\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:26:32.247717 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:32.247672 5935 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8f89894d-a27e-44f9-8e7a-f1c909500cd4-metrics-certs\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:26:32.247717 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:32.247688 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhsqg\" (UniqueName: \"kubernetes.io/projected/8f89894d-a27e-44f9-8e7a-f1c909500cd4-kube-api-access-xhsqg\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:26:32.247717 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:32.247700 5935 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8f89894d-a27e-44f9-8e7a-f1c909500cd4-maas-api-tls\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:26:32.276231 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:32.276196 5935 generic.go:356] "Generic (PLEG): container finished" podID="8f89894d-a27e-44f9-8e7a-f1c909500cd4" containerID="8fb148b7ab9cd1e1e84aca6411e8baccc53f2471777bf0a01eb5679b450d38a9" exitCode=0 Nov 01 20:26:32.276403 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:32.276259 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2babaef6-77b4b5bd9b-rzgmf" Nov 01 20:26:32.276403 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:32.276281 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2babaef6-77b4b5bd9b-rzgmf" event={"ID":"8f89894d-a27e-44f9-8e7a-f1c909500cd4","Type":"ContainerDied","Data":"8fb148b7ab9cd1e1e84aca6411e8baccc53f2471777bf0a01eb5679b450d38a9"} Nov 01 20:26:32.276403 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:32.276381 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2babaef6-77b4b5bd9b-rzgmf" event={"ID":"8f89894d-a27e-44f9-8e7a-f1c909500cd4","Type":"ContainerDied","Data":"6ae694c02b34f0455a4e80fd8f3cf34f7ab7c38ab49355ed1552131a1e70cadd"} Nov 01 20:26:32.276572 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:32.276409 5935 scope.go:122] "RemoveContainer" containerID="8fb148b7ab9cd1e1e84aca6411e8baccc53f2471777bf0a01eb5679b450d38a9" Nov 01 20:26:32.288964 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:32.288947 5935 scope.go:122] "RemoveContainer" containerID="8fb148b7ab9cd1e1e84aca6411e8baccc53f2471777bf0a01eb5679b450d38a9" Nov 01 20:26:32.289247 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:26:32.289221 5935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fb148b7ab9cd1e1e84aca6411e8baccc53f2471777bf0a01eb5679b450d38a9\": container with ID starting with 8fb148b7ab9cd1e1e84aca6411e8baccc53f2471777bf0a01eb5679b450d38a9 not found: ID does not exist" containerID="8fb148b7ab9cd1e1e84aca6411e8baccc53f2471777bf0a01eb5679b450d38a9" Nov 01 20:26:32.289333 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:32.289257 5935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fb148b7ab9cd1e1e84aca6411e8baccc53f2471777bf0a01eb5679b450d38a9"} err="failed to get container status \"8fb148b7ab9cd1e1e84aca6411e8baccc53f2471777bf0a01eb5679b450d38a9\": rpc error: code = NotFound desc = could not find container \"8fb148b7ab9cd1e1e84aca6411e8baccc53f2471777bf0a01eb5679b450d38a9\": container with ID starting with 8fb148b7ab9cd1e1e84aca6411e8baccc53f2471777bf0a01eb5679b450d38a9 not found: ID does not exist" Nov 01 20:26:32.300022 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:32.299993 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2babaef6-77b4b5bd9b-rzgmf"] Nov 01 20:26:32.308857 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:32.308837 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2babaef6-77b4b5bd9b-rzgmf"] Nov 01 20:26:32.353317 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:32.353281 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f89894d-a27e-44f9-8e7a-f1c909500cd4" path="/var/lib/kubelet/pods/8f89894d-a27e-44f9-8e7a-f1c909500cd4/volumes" Nov 01 20:26:35.844852 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:26:35.844804 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:26:41.212993 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:41.212959 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-8b6cd6/sub-test-model-8b6cd6-kserve-6c44c95575-sg7gm"] Nov 01 20:26:41.213864 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:41.213842 5935 cpu_manager.go:414] "containerMap: removing container" podUID="86e63000-fc7e-4429-af13-4046198bca02" containerName="istio-proxy" Nov 01 20:26:41.213948 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:41.213867 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86e63000-fc7e-4429-af13-4046198bca02" containerName="istio-proxy" Nov 01 20:26:41.213948 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:41.213896 5935 cpu_manager.go:414] "containerMap: removing container" podUID="098cc87f-f041-4d32-b909-be5a16ced4d8" containerName="main" Nov 01 20:26:41.213948 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:41.213904 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="098cc87f-f041-4d32-b909-be5a16ced4d8" containerName="main" Nov 01 20:26:41.213948 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:41.213934 5935 cpu_manager.go:414] "containerMap: removing container" podUID="b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae" containerName="istio-proxy" Nov 01 20:26:41.213948 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:41.213943 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae" containerName="istio-proxy" Nov 01 20:26:41.214196 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:41.213972 5935 cpu_manager.go:414] "containerMap: removing container" podUID="16f53549-0a34-4825-a719-7ce0eff818ac" containerName="istio-proxy" Nov 01 20:26:41.214196 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:41.213979 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16f53549-0a34-4825-a719-7ce0eff818ac" containerName="istio-proxy" Nov 01 20:26:41.214196 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:41.213994 5935 cpu_manager.go:414] "containerMap: removing container" podUID="f8249130-8b25-468a-9904-2564d38fdb15" containerName="istio-proxy" Nov 01 20:26:41.214196 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:41.214001 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8249130-8b25-468a-9904-2564d38fdb15" containerName="istio-proxy" Nov 01 20:26:41.214196 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:41.214011 5935 cpu_manager.go:414] "containerMap: removing container" podUID="8f89894d-a27e-44f9-8e7a-f1c909500cd4" containerName="maas-api" Nov 01 20:26:41.214196 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:41.214017 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f89894d-a27e-44f9-8e7a-f1c909500cd4" containerName="maas-api" Nov 01 20:26:41.214196 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:41.214136 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="86e63000-fc7e-4429-af13-4046198bca02" containerName="istio-proxy" Nov 01 20:26:41.214196 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:41.214152 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="098cc87f-f041-4d32-b909-be5a16ced4d8" containerName="main" Nov 01 20:26:41.214196 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:41.214164 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="b52dfb6f-d9c7-4b1f-9cca-13d67c4758ae" containerName="istio-proxy" Nov 01 20:26:41.214196 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:41.214175 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="16f53549-0a34-4825-a719-7ce0eff818ac" containerName="istio-proxy" Nov 01 20:26:41.214196 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:41.214185 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8249130-8b25-468a-9904-2564d38fdb15" containerName="istio-proxy" Nov 01 20:26:41.214196 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:41.214192 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f89894d-a27e-44f9-8e7a-f1c909500cd4" containerName="maas-api" Nov 01 20:26:41.243568 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:41.243529 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-8b6cd6/sub-test-model-8b6cd6-kserve-6c44c95575-sg7gm"] Nov 01 20:26:41.243736 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:41.243654 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-8b6cd6/sub-test-model-8b6cd6-kserve-6c44c95575-sg7gm" Nov 01 20:26:41.248073 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:41.248048 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-8b6cd6\"/\"openshift-service-ca.crt\"" Nov 01 20:26:41.248497 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:41.248088 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-8b6cd6\"/\"default-dockercfg-5zn99\"" Nov 01 20:26:41.248497 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:41.248251 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-8b6cd6\"/\"kube-root-ca.crt\"" Nov 01 20:26:41.248497 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:41.248346 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-8b6cd6\"/\"sub-test-model-8b6cd6-kserve-self-signed-certs\"" Nov 01 20:26:41.391720 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:41.391682 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50eb270e-173d-4727-9504-1b69379f7f77-dshm\") pod \"sub-test-model-8b6cd6-kserve-6c44c95575-sg7gm\" (UID: \"50eb270e-173d-4727-9504-1b69379f7f77\") " pod="ai-tenant-e2e-shared-a-w4-8b6cd6/sub-test-model-8b6cd6-kserve-6c44c95575-sg7gm" Nov 01 20:26:41.391865 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:41.391788 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50eb270e-173d-4727-9504-1b69379f7f77-tls-certs\") pod \"sub-test-model-8b6cd6-kserve-6c44c95575-sg7gm\" (UID: \"50eb270e-173d-4727-9504-1b69379f7f77\") " pod="ai-tenant-e2e-shared-a-w4-8b6cd6/sub-test-model-8b6cd6-kserve-6c44c95575-sg7gm" Nov 01 20:26:41.391865 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:41.391842 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50eb270e-173d-4727-9504-1b69379f7f77-model-cache\") pod \"sub-test-model-8b6cd6-kserve-6c44c95575-sg7gm\" (UID: \"50eb270e-173d-4727-9504-1b69379f7f77\") " pod="ai-tenant-e2e-shared-a-w4-8b6cd6/sub-test-model-8b6cd6-kserve-6c44c95575-sg7gm" Nov 01 20:26:41.391952 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:41.391934 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50eb270e-173d-4727-9504-1b69379f7f77-home\") pod \"sub-test-model-8b6cd6-kserve-6c44c95575-sg7gm\" (UID: \"50eb270e-173d-4727-9504-1b69379f7f77\") " pod="ai-tenant-e2e-shared-a-w4-8b6cd6/sub-test-model-8b6cd6-kserve-6c44c95575-sg7gm" Nov 01 20:26:41.392085 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:41.392063 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50eb270e-173d-4727-9504-1b69379f7f77-tmp-dir\") pod \"sub-test-model-8b6cd6-kserve-6c44c95575-sg7gm\" (UID: \"50eb270e-173d-4727-9504-1b69379f7f77\") " pod="ai-tenant-e2e-shared-a-w4-8b6cd6/sub-test-model-8b6cd6-kserve-6c44c95575-sg7gm" Nov 01 20:26:41.392120 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:41.392107 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48tjm\" (UniqueName: \"kubernetes.io/projected/50eb270e-173d-4727-9504-1b69379f7f77-kube-api-access-48tjm\") pod \"sub-test-model-8b6cd6-kserve-6c44c95575-sg7gm\" (UID: \"50eb270e-173d-4727-9504-1b69379f7f77\") " pod="ai-tenant-e2e-shared-a-w4-8b6cd6/sub-test-model-8b6cd6-kserve-6c44c95575-sg7gm" Nov 01 20:26:41.493825 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:41.493735 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50eb270e-173d-4727-9504-1b69379f7f77-tls-certs\") pod \"sub-test-model-8b6cd6-kserve-6c44c95575-sg7gm\" (UID: \"50eb270e-173d-4727-9504-1b69379f7f77\") " pod="ai-tenant-e2e-shared-a-w4-8b6cd6/sub-test-model-8b6cd6-kserve-6c44c95575-sg7gm" Nov 01 20:26:41.494012 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:41.493904 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50eb270e-173d-4727-9504-1b69379f7f77-model-cache\") pod \"sub-test-model-8b6cd6-kserve-6c44c95575-sg7gm\" (UID: \"50eb270e-173d-4727-9504-1b69379f7f77\") " pod="ai-tenant-e2e-shared-a-w4-8b6cd6/sub-test-model-8b6cd6-kserve-6c44c95575-sg7gm" Nov 01 20:26:41.494012 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:41.493999 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50eb270e-173d-4727-9504-1b69379f7f77-home\") pod \"sub-test-model-8b6cd6-kserve-6c44c95575-sg7gm\" (UID: \"50eb270e-173d-4727-9504-1b69379f7f77\") " pod="ai-tenant-e2e-shared-a-w4-8b6cd6/sub-test-model-8b6cd6-kserve-6c44c95575-sg7gm" Nov 01 20:26:41.494236 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:41.494199 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50eb270e-173d-4727-9504-1b69379f7f77-tmp-dir\") pod \"sub-test-model-8b6cd6-kserve-6c44c95575-sg7gm\" (UID: \"50eb270e-173d-4727-9504-1b69379f7f77\") " pod="ai-tenant-e2e-shared-a-w4-8b6cd6/sub-test-model-8b6cd6-kserve-6c44c95575-sg7gm" Nov 01 20:26:41.494314 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:41.494254 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48tjm\" (UniqueName: \"kubernetes.io/projected/50eb270e-173d-4727-9504-1b69379f7f77-kube-api-access-48tjm\") pod \"sub-test-model-8b6cd6-kserve-6c44c95575-sg7gm\" (UID: \"50eb270e-173d-4727-9504-1b69379f7f77\") " pod="ai-tenant-e2e-shared-a-w4-8b6cd6/sub-test-model-8b6cd6-kserve-6c44c95575-sg7gm" Nov 01 20:26:41.494428 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:41.494408 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50eb270e-173d-4727-9504-1b69379f7f77-dshm\") pod \"sub-test-model-8b6cd6-kserve-6c44c95575-sg7gm\" (UID: \"50eb270e-173d-4727-9504-1b69379f7f77\") " pod="ai-tenant-e2e-shared-a-w4-8b6cd6/sub-test-model-8b6cd6-kserve-6c44c95575-sg7gm" Nov 01 20:26:41.494504 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:41.494457 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50eb270e-173d-4727-9504-1b69379f7f77-model-cache\") pod \"sub-test-model-8b6cd6-kserve-6c44c95575-sg7gm\" (UID: \"50eb270e-173d-4727-9504-1b69379f7f77\") " pod="ai-tenant-e2e-shared-a-w4-8b6cd6/sub-test-model-8b6cd6-kserve-6c44c95575-sg7gm" Nov 01 20:26:41.494557 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:41.494510 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50eb270e-173d-4727-9504-1b69379f7f77-home\") pod \"sub-test-model-8b6cd6-kserve-6c44c95575-sg7gm\" (UID: \"50eb270e-173d-4727-9504-1b69379f7f77\") " pod="ai-tenant-e2e-shared-a-w4-8b6cd6/sub-test-model-8b6cd6-kserve-6c44c95575-sg7gm" Nov 01 20:26:41.494998 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:41.494976 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50eb270e-173d-4727-9504-1b69379f7f77-tmp-dir\") pod \"sub-test-model-8b6cd6-kserve-6c44c95575-sg7gm\" (UID: \"50eb270e-173d-4727-9504-1b69379f7f77\") " pod="ai-tenant-e2e-shared-a-w4-8b6cd6/sub-test-model-8b6cd6-kserve-6c44c95575-sg7gm" Nov 01 20:26:41.496695 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:41.496671 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50eb270e-173d-4727-9504-1b69379f7f77-tls-certs\") pod \"sub-test-model-8b6cd6-kserve-6c44c95575-sg7gm\" (UID: \"50eb270e-173d-4727-9504-1b69379f7f77\") " pod="ai-tenant-e2e-shared-a-w4-8b6cd6/sub-test-model-8b6cd6-kserve-6c44c95575-sg7gm" Nov 01 20:26:41.497528 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:41.497503 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50eb270e-173d-4727-9504-1b69379f7f77-dshm\") pod \"sub-test-model-8b6cd6-kserve-6c44c95575-sg7gm\" (UID: \"50eb270e-173d-4727-9504-1b69379f7f77\") " pod="ai-tenant-e2e-shared-a-w4-8b6cd6/sub-test-model-8b6cd6-kserve-6c44c95575-sg7gm" Nov 01 20:26:41.504256 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:41.504237 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48tjm\" (UniqueName: \"kubernetes.io/projected/50eb270e-173d-4727-9504-1b69379f7f77-kube-api-access-48tjm\") pod \"sub-test-model-8b6cd6-kserve-6c44c95575-sg7gm\" (UID: \"50eb270e-173d-4727-9504-1b69379f7f77\") " pod="ai-tenant-e2e-shared-a-w4-8b6cd6/sub-test-model-8b6cd6-kserve-6c44c95575-sg7gm" Nov 01 20:26:41.560822 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:41.560794 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-8b6cd6/sub-test-model-8b6cd6-kserve-6c44c95575-sg7gm" Nov 01 20:26:41.839276 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:41.839243 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-8b6cd6/sub-test-model-8b6cd6-kserve-6c44c95575-sg7gm"] Nov 01 20:26:41.839952 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:26:41.839922 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod50eb270e_173d_4727_9504_1b69379f7f77.slice/crio-d3bbb0f43af74448e9c6678284745135247743779828964e850f167dd2c8677c WatchSource:0}: Error finding container d3bbb0f43af74448e9c6678284745135247743779828964e850f167dd2c8677c: Status 404 returned error can't find the container with id d3bbb0f43af74448e9c6678284745135247743779828964e850f167dd2c8677c Nov 01 20:26:42.267732 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:42.267694 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-9ea5d2/payload-processing-external-model-e2e-extmod-b-9ea5d2-86f8rg9hq"] Nov 01 20:26:42.272271 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:42.272242 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-9ea5d2/payload-processing-external-model-e2e-extmod-b-9ea5d2-86f8rg9hq" Nov 01 20:26:42.281102 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:42.281065 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-9ea5d2\"/\"agc-cred-d46acf38634e\"" Nov 01 20:26:42.281102 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:42.281091 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-9ea5d2\"/\"kube-root-ca.crt\"" Nov 01 20:26:42.281322 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:42.281092 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-9ea5d2\"/\"openshift-service-ca.crt\"" Nov 01 20:26:42.281322 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:42.281129 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-9ea5d2\"/\"routing-overlay\"" Nov 01 20:26:42.281322 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:42.281172 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-9ea5d2\"/\"payload-processing-external-model-plugins-e2e-extmod-b-9ea5d2\"" Nov 01 20:26:42.281322 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:42.281184 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-9ea5d2\"/\"payload-processing-external-model-e2e--52260de0-dockercfg-hrnk9\"" Nov 01 20:26:42.283735 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:42.283711 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-9ea5d2/payload-processing-external-model-e2e-extmod-b-9ea5d2-86f8rg9hq"] Nov 01 20:26:42.387069 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:42.387032 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-8b6cd6/sub-test-model-8b6cd6-kserve-6c44c95575-sg7gm" event={"ID":"50eb270e-173d-4727-9504-1b69379f7f77","Type":"ContainerStarted","Data":"1b245a89336d668a1bd81e88ecdb0579b4a98843928f28d2c12aa068f69f1f3b"} Nov 01 20:26:42.387069 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:42.387083 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-8b6cd6/sub-test-model-8b6cd6-kserve-6c44c95575-sg7gm" event={"ID":"50eb270e-173d-4727-9504-1b69379f7f77","Type":"ContainerStarted","Data":"d3bbb0f43af74448e9c6678284745135247743779828964e850f167dd2c8677c"} Nov 01 20:26:42.421311 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:42.412546 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-8b6cd6/sub-test-model-8b6cd6-kserve-6c44c95575-sg7gm" podStartSLOduration=1.412526437 podStartE2EDuration="1.412526437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:26:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:26:42.411824639 +0000 UTC m=+1648.184985158" watchObservedRunningTime="2026-11-01 20:26:42.412526437 +0000 UTC m=+1648.185686950" Nov 01 20:26:42.436307 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:42.422068 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/436a09cc-49bc-4784-a4a1-4e36190a3e6a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-9ea5d2-86f8rg9hq\" (UID: \"436a09cc-49bc-4784-a4a1-4e36190a3e6a\") " pod="ai-tenant-e2e-extmod-b-9ea5d2/payload-processing-external-model-e2e-extmod-b-9ea5d2-86f8rg9hq" Nov 01 20:26:42.436307 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:42.422151 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/436a09cc-49bc-4784-a4a1-4e36190a3e6a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-9ea5d2-86f8rg9hq\" (UID: \"436a09cc-49bc-4784-a4a1-4e36190a3e6a\") " pod="ai-tenant-e2e-extmod-b-9ea5d2/payload-processing-external-model-e2e-extmod-b-9ea5d2-86f8rg9hq" Nov 01 20:26:42.436307 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:42.422236 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzvfz\" (UniqueName: \"kubernetes.io/projected/436a09cc-49bc-4784-a4a1-4e36190a3e6a-kube-api-access-bzvfz\") pod \"payload-processing-external-model-e2e-extmod-b-9ea5d2-86f8rg9hq\" (UID: \"436a09cc-49bc-4784-a4a1-4e36190a3e6a\") " pod="ai-tenant-e2e-extmod-b-9ea5d2/payload-processing-external-model-e2e-extmod-b-9ea5d2-86f8rg9hq" Nov 01 20:26:42.436307 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:42.422277 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/436a09cc-49bc-4784-a4a1-4e36190a3e6a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-9ea5d2-86f8rg9hq\" (UID: \"436a09cc-49bc-4784-a4a1-4e36190a3e6a\") " pod="ai-tenant-e2e-extmod-b-9ea5d2/payload-processing-external-model-e2e-extmod-b-9ea5d2-86f8rg9hq" Nov 01 20:26:42.523759 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:42.523675 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzvfz\" (UniqueName: \"kubernetes.io/projected/436a09cc-49bc-4784-a4a1-4e36190a3e6a-kube-api-access-bzvfz\") pod \"payload-processing-external-model-e2e-extmod-b-9ea5d2-86f8rg9hq\" (UID: \"436a09cc-49bc-4784-a4a1-4e36190a3e6a\") " pod="ai-tenant-e2e-extmod-b-9ea5d2/payload-processing-external-model-e2e-extmod-b-9ea5d2-86f8rg9hq" Nov 01 20:26:42.524387 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:42.524357 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/436a09cc-49bc-4784-a4a1-4e36190a3e6a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-9ea5d2-86f8rg9hq\" (UID: \"436a09cc-49bc-4784-a4a1-4e36190a3e6a\") " pod="ai-tenant-e2e-extmod-b-9ea5d2/payload-processing-external-model-e2e-extmod-b-9ea5d2-86f8rg9hq" Nov 01 20:26:42.524728 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:42.524705 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/436a09cc-49bc-4784-a4a1-4e36190a3e6a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-9ea5d2-86f8rg9hq\" (UID: \"436a09cc-49bc-4784-a4a1-4e36190a3e6a\") " pod="ai-tenant-e2e-extmod-b-9ea5d2/payload-processing-external-model-e2e-extmod-b-9ea5d2-86f8rg9hq" Nov 01 20:26:42.524919 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:42.524900 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/436a09cc-49bc-4784-a4a1-4e36190a3e6a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-9ea5d2-86f8rg9hq\" (UID: \"436a09cc-49bc-4784-a4a1-4e36190a3e6a\") " pod="ai-tenant-e2e-extmod-b-9ea5d2/payload-processing-external-model-e2e-extmod-b-9ea5d2-86f8rg9hq" Nov 01 20:26:42.526014 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:42.525985 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/436a09cc-49bc-4784-a4a1-4e36190a3e6a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-9ea5d2-86f8rg9hq\" (UID: \"436a09cc-49bc-4784-a4a1-4e36190a3e6a\") " pod="ai-tenant-e2e-extmod-b-9ea5d2/payload-processing-external-model-e2e-extmod-b-9ea5d2-86f8rg9hq" Nov 01 20:26:42.526711 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:42.526692 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/436a09cc-49bc-4784-a4a1-4e36190a3e6a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-9ea5d2-86f8rg9hq\" (UID: \"436a09cc-49bc-4784-a4a1-4e36190a3e6a\") " pod="ai-tenant-e2e-extmod-b-9ea5d2/payload-processing-external-model-e2e-extmod-b-9ea5d2-86f8rg9hq" Nov 01 20:26:42.529431 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:42.529076 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/436a09cc-49bc-4784-a4a1-4e36190a3e6a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-9ea5d2-86f8rg9hq\" (UID: \"436a09cc-49bc-4784-a4a1-4e36190a3e6a\") " pod="ai-tenant-e2e-extmod-b-9ea5d2/payload-processing-external-model-e2e-extmod-b-9ea5d2-86f8rg9hq" Nov 01 20:26:42.535058 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:42.535025 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzvfz\" (UniqueName: \"kubernetes.io/projected/436a09cc-49bc-4784-a4a1-4e36190a3e6a-kube-api-access-bzvfz\") pod \"payload-processing-external-model-e2e-extmod-b-9ea5d2-86f8rg9hq\" (UID: \"436a09cc-49bc-4784-a4a1-4e36190a3e6a\") " pod="ai-tenant-e2e-extmod-b-9ea5d2/payload-processing-external-model-e2e-extmod-b-9ea5d2-86f8rg9hq" Nov 01 20:26:42.561612 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:42.561584 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-8b6cd6/sub-test-model-8b6cd6-kserve-6c44c95575-sg7gm" Nov 01 20:26:42.588599 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:42.588569 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-9ea5d2/payload-processing-external-model-e2e-extmod-b-9ea5d2-86f8rg9hq" Nov 01 20:26:42.858570 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:42.858494 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd"] Nov 01 20:26:42.858848 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:42.858730 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd" podUID="8686280e-385b-4a98-a1d2-5018e5f08fc1" containerName="istio-proxy" containerID="cri-o://186f5408970ca77b454a3ad7ec7b60100fe1e84fd96e75d6232dad91ab56ba9e" gracePeriod=30 Nov 01 20:26:42.896872 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:42.896840 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-9ea5d2/payload-processing-external-model-e2e-extmod-b-9ea5d2-86f8rg9hq"] Nov 01 20:26:42.902044 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:26:42.902018 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod436a09cc_49bc_4784_a4a1_4e36190a3e6a.slice/crio-0ae2bb016ee461cd4230d6063d879a7fb78a987e7574267f109aada2b1862abd WatchSource:0}: Error finding container 0ae2bb016ee461cd4230d6063d879a7fb78a987e7574267f109aada2b1862abd: Status 404 returned error can't find the container with id 0ae2bb016ee461cd4230d6063d879a7fb78a987e7574267f109aada2b1862abd Nov 01 20:26:43.395513 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:43.395476 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-9ea5d2/payload-processing-external-model-e2e-extmod-b-9ea5d2-86f8rg9hq" event={"ID":"436a09cc-49bc-4784-a4a1-4e36190a3e6a","Type":"ContainerStarted","Data":"24b66071f7a01df256514931a35da71ec034c9390afe00a66ad584dfc9afa139"} Nov 01 20:26:43.395940 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:43.395546 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-9ea5d2/payload-processing-external-model-e2e-extmod-b-9ea5d2-86f8rg9hq" Nov 01 20:26:43.395940 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:43.395572 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-9ea5d2/payload-processing-external-model-e2e-extmod-b-9ea5d2-86f8rg9hq" event={"ID":"436a09cc-49bc-4784-a4a1-4e36190a3e6a","Type":"ContainerStarted","Data":"0ae2bb016ee461cd4230d6063d879a7fb78a987e7574267f109aada2b1862abd"} Nov 01 20:26:43.417864 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:43.417810 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-9ea5d2/payload-processing-external-model-e2e-extmod-b-9ea5d2-86f8rg9hq" podStartSLOduration=1.417781522 podStartE2EDuration="1.417781522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:26:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:26:43.413809018 +0000 UTC m=+1649.186969531" watchObservedRunningTime="2026-11-01 20:26:43.417781522 +0000 UTC m=+1649.190942034" Nov 01 20:26:48.118666 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:48.118639 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd" Nov 01 20:26:48.193125 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:48.193096 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8686280e-385b-4a98-a1d2-5018e5f08fc1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8686280e-385b-4a98-a1d2-5018e5f08fc1-istio-data\") pod \"8686280e-385b-4a98-a1d2-5018e5f08fc1\" (UID: \"8686280e-385b-4a98-a1d2-5018e5f08fc1\") " Nov 01 20:26:48.193279 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:48.193165 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8686280e-385b-4a98-a1d2-5018e5f08fc1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8686280e-385b-4a98-a1d2-5018e5f08fc1-workload-socket\") pod \"8686280e-385b-4a98-a1d2-5018e5f08fc1\" (UID: \"8686280e-385b-4a98-a1d2-5018e5f08fc1\") " Nov 01 20:26:48.193279 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:48.193227 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8686280e-385b-4a98-a1d2-5018e5f08fc1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8686280e-385b-4a98-a1d2-5018e5f08fc1-istiod-ca-cert\") pod \"8686280e-385b-4a98-a1d2-5018e5f08fc1\" (UID: \"8686280e-385b-4a98-a1d2-5018e5f08fc1\") " Nov 01 20:26:48.193412 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:48.193395 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8686280e-385b-4a98-a1d2-5018e5f08fc1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8686280e-385b-4a98-a1d2-5018e5f08fc1-istio-envoy\") pod \"8686280e-385b-4a98-a1d2-5018e5f08fc1\" (UID: \"8686280e-385b-4a98-a1d2-5018e5f08fc1\") " Nov 01 20:26:48.193452 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:48.193438 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8686280e-385b-4a98-a1d2-5018e5f08fc1-istio-data" pod "8686280e-385b-4a98-a1d2-5018e5f08fc1" (UID: "8686280e-385b-4a98-a1d2-5018e5f08fc1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:26:48.193491 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:48.193439 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8686280e-385b-4a98-a1d2-5018e5f08fc1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8686280e-385b-4a98-a1d2-5018e5f08fc1-credential-socket\") pod \"8686280e-385b-4a98-a1d2-5018e5f08fc1\" (UID: \"8686280e-385b-4a98-a1d2-5018e5f08fc1\") " Nov 01 20:26:48.193531 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:48.193478 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8686280e-385b-4a98-a1d2-5018e5f08fc1-workload-socket" pod "8686280e-385b-4a98-a1d2-5018e5f08fc1" (UID: "8686280e-385b-4a98-a1d2-5018e5f08fc1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:26:48.193569 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:48.193528 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8686280e-385b-4a98-a1d2-5018e5f08fc1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8686280e-385b-4a98-a1d2-5018e5f08fc1-istio-podinfo\") pod \"8686280e-385b-4a98-a1d2-5018e5f08fc1\" (UID: \"8686280e-385b-4a98-a1d2-5018e5f08fc1\") " Nov 01 20:26:48.193605 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:48.193565 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8686280e-385b-4a98-a1d2-5018e5f08fc1-istio-token\" (UniqueName: \"kubernetes.io/projected/8686280e-385b-4a98-a1d2-5018e5f08fc1-istio-token\") pod \"8686280e-385b-4a98-a1d2-5018e5f08fc1\" (UID: \"8686280e-385b-4a98-a1d2-5018e5f08fc1\") " Nov 01 20:26:48.193605 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:48.193596 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8686280e-385b-4a98-a1d2-5018e5f08fc1-kube-api-access-dn86x\" (UniqueName: \"kubernetes.io/projected/8686280e-385b-4a98-a1d2-5018e5f08fc1-kube-api-access-dn86x\") pod \"8686280e-385b-4a98-a1d2-5018e5f08fc1\" (UID: \"8686280e-385b-4a98-a1d2-5018e5f08fc1\") " Nov 01 20:26:48.193683 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:48.193628 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8686280e-385b-4a98-a1d2-5018e5f08fc1-credential-socket" pod "8686280e-385b-4a98-a1d2-5018e5f08fc1" (UID: "8686280e-385b-4a98-a1d2-5018e5f08fc1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:26:48.193683 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:48.193637 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8686280e-385b-4a98-a1d2-5018e5f08fc1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8686280e-385b-4a98-a1d2-5018e5f08fc1-workload-certs\") pod \"8686280e-385b-4a98-a1d2-5018e5f08fc1\" (UID: \"8686280e-385b-4a98-a1d2-5018e5f08fc1\") " Nov 01 20:26:48.193683 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:48.193648 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8686280e-385b-4a98-a1d2-5018e5f08fc1-istiod-ca-cert" pod "8686280e-385b-4a98-a1d2-5018e5f08fc1" (UID: "8686280e-385b-4a98-a1d2-5018e5f08fc1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:26:48.193965 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:48.193938 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8686280e-385b-4a98-a1d2-5018e5f08fc1-workload-certs" pod "8686280e-385b-4a98-a1d2-5018e5f08fc1" (UID: "8686280e-385b-4a98-a1d2-5018e5f08fc1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:26:48.194096 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:48.194077 5935 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8686280e-385b-4a98-a1d2-5018e5f08fc1-credential-socket\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:26:48.194138 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:48.194104 5935 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8686280e-385b-4a98-a1d2-5018e5f08fc1-workload-certs\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:26:48.194138 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:48.194119 5935 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8686280e-385b-4a98-a1d2-5018e5f08fc1-istio-data\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:26:48.194138 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:48.194133 5935 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8686280e-385b-4a98-a1d2-5018e5f08fc1-workload-socket\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:26:48.194256 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:48.194146 5935 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8686280e-385b-4a98-a1d2-5018e5f08fc1-istiod-ca-cert\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:26:48.195612 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:48.195591 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8686280e-385b-4a98-a1d2-5018e5f08fc1-istio-envoy" pod "8686280e-385b-4a98-a1d2-5018e5f08fc1" (UID: "8686280e-385b-4a98-a1d2-5018e5f08fc1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:26:48.195726 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:48.195682 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8686280e-385b-4a98-a1d2-5018e5f08fc1-kube-api-access-dn86x" pod "8686280e-385b-4a98-a1d2-5018e5f08fc1" (UID: "8686280e-385b-4a98-a1d2-5018e5f08fc1"). InnerVolumeSpecName "kube-api-access-dn86x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:26:48.195975 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:48.195959 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8686280e-385b-4a98-a1d2-5018e5f08fc1-istio-token" pod "8686280e-385b-4a98-a1d2-5018e5f08fc1" (UID: "8686280e-385b-4a98-a1d2-5018e5f08fc1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:26:48.196044 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:48.195962 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8686280e-385b-4a98-a1d2-5018e5f08fc1-istio-podinfo" pod "8686280e-385b-4a98-a1d2-5018e5f08fc1" (UID: "8686280e-385b-4a98-a1d2-5018e5f08fc1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:26:48.296434 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:48.296401 5935 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8686280e-385b-4a98-a1d2-5018e5f08fc1-istio-podinfo\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:26:48.296434 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:48.296433 5935 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8686280e-385b-4a98-a1d2-5018e5f08fc1-istio-token\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:26:48.297620 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:48.296450 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dn86x\" (UniqueName: \"kubernetes.io/projected/8686280e-385b-4a98-a1d2-5018e5f08fc1-kube-api-access-dn86x\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:26:48.297620 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:48.296464 5935 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8686280e-385b-4a98-a1d2-5018e5f08fc1-istio-envoy\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:26:48.453176 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:48.453100 5935 generic.go:356] "Generic (PLEG): container finished" podID="8686280e-385b-4a98-a1d2-5018e5f08fc1" containerID="186f5408970ca77b454a3ad7ec7b60100fe1e84fd96e75d6232dad91ab56ba9e" exitCode=0 Nov 01 20:26:48.453176 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:48.453139 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd" Nov 01 20:26:48.453369 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:48.453189 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd" event={"ID":"8686280e-385b-4a98-a1d2-5018e5f08fc1","Type":"ContainerDied","Data":"186f5408970ca77b454a3ad7ec7b60100fe1e84fd96e75d6232dad91ab56ba9e"} Nov 01 20:26:48.453369 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:48.453232 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd" event={"ID":"8686280e-385b-4a98-a1d2-5018e5f08fc1","Type":"ContainerDied","Data":"377a07338d3eb72d60b2dd38594e9b157855383457987860454957d1b548050b"} Nov 01 20:26:48.453369 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:48.453254 5935 scope.go:122] "RemoveContainer" containerID="186f5408970ca77b454a3ad7ec7b60100fe1e84fd96e75d6232dad91ab56ba9e" Nov 01 20:26:48.463722 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:48.463700 5935 scope.go:122] "RemoveContainer" containerID="186f5408970ca77b454a3ad7ec7b60100fe1e84fd96e75d6232dad91ab56ba9e" Nov 01 20:26:48.463974 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:26:48.463955 5935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"186f5408970ca77b454a3ad7ec7b60100fe1e84fd96e75d6232dad91ab56ba9e\": container with ID starting with 186f5408970ca77b454a3ad7ec7b60100fe1e84fd96e75d6232dad91ab56ba9e not found: ID does not exist" containerID="186f5408970ca77b454a3ad7ec7b60100fe1e84fd96e75d6232dad91ab56ba9e" Nov 01 20:26:48.464024 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:48.463980 5935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"186f5408970ca77b454a3ad7ec7b60100fe1e84fd96e75d6232dad91ab56ba9e"} err="failed to get container status \"186f5408970ca77b454a3ad7ec7b60100fe1e84fd96e75d6232dad91ab56ba9e\": rpc error: code = NotFound desc = could not find container \"186f5408970ca77b454a3ad7ec7b60100fe1e84fd96e75d6232dad91ab56ba9e\": container with ID starting with 186f5408970ca77b454a3ad7ec7b60100fe1e84fd96e75d6232dad91ab56ba9e not found: ID does not exist" Nov 01 20:26:48.471654 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:48.471633 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd"] Nov 01 20:26:48.477888 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:48.477866 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-pfcrd"] Nov 01 20:26:50.356769 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:50.356734 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8686280e-385b-4a98-a1d2-5018e5f08fc1" path="/var/lib/kubelet/pods/8686280e-385b-4a98-a1d2-5018e5f08fc1/volumes" Nov 01 20:26:51.561309 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:51.561252 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-8b6cd6/sub-test-model-8b6cd6-kserve-6c44c95575-sg7gm" Nov 01 20:26:51.573461 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:51.573438 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-8b6cd6/sub-test-model-8b6cd6-kserve-6c44c95575-sg7gm" Nov 01 20:26:52.490794 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:52.490760 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-8b6cd6/sub-test-model-8b6cd6-kserve-6c44c95575-sg7gm" Nov 01 20:26:54.434245 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:26:54.434218 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-9ea5d2/payload-processing-external-model-e2e-extmod-b-9ea5d2-86f8rg9hq" Nov 01 20:27:05.843462 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:27:05.843425 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:27:35.843869 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:27:35.843833 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:27:39.306380 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:27:39.306347 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-9ea5d2/payload-processing-external-model-e2e-extmod-b-9ea5d2-86f8rg9hq"] Nov 01 20:27:44.307364 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:27:44.307322 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-9ea5d2/payload-processing-external-model-e2e-extmod-b-9ea5d2-86f8rg9hq" podUID="436a09cc-49bc-4784-a4a1-4e36190a3e6a" containerName="payload-processing" containerID="cri-o://24b66071f7a01df256514931a35da71ec034c9390afe00a66ad584dfc9afa139" gracePeriod=25 Nov 01 20:27:44.415799 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:27:44.415772 5935 logging.go:55] [core] [Channel #2494 SubChannel #2495]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.111:50052", ServerName: "10.133.0.111:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.111:50052: connect: connection refused" Nov 01 20:27:44.565055 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:27:44.564998 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-9ea5d2/payload-processing-external-model-e2e-extmod-b-9ea5d2-86f8rg9hq" Nov 01 20:27:44.631094 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:27:44.631063 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/436a09cc-49bc-4784-a4a1-4e36190a3e6a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/436a09cc-49bc-4784-a4a1-4e36190a3e6a-routing-overlay-volume\") pod \"436a09cc-49bc-4784-a4a1-4e36190a3e6a\" (UID: \"436a09cc-49bc-4784-a4a1-4e36190a3e6a\") " Nov 01 20:27:44.631268 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:27:44.631106 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/436a09cc-49bc-4784-a4a1-4e36190a3e6a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/436a09cc-49bc-4784-a4a1-4e36190a3e6a-plugins-config-volume\") pod \"436a09cc-49bc-4784-a4a1-4e36190a3e6a\" (UID: \"436a09cc-49bc-4784-a4a1-4e36190a3e6a\") " Nov 01 20:27:44.631268 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:27:44.631231 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/436a09cc-49bc-4784-a4a1-4e36190a3e6a-kube-api-access-bzvfz\" (UniqueName: \"kubernetes.io/projected/436a09cc-49bc-4784-a4a1-4e36190a3e6a-kube-api-access-bzvfz\") pod \"436a09cc-49bc-4784-a4a1-4e36190a3e6a\" (UID: \"436a09cc-49bc-4784-a4a1-4e36190a3e6a\") " Nov 01 20:27:44.631268 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:27:44.631263 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/436a09cc-49bc-4784-a4a1-4e36190a3e6a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/436a09cc-49bc-4784-a4a1-4e36190a3e6a-provider-credentials\") pod \"436a09cc-49bc-4784-a4a1-4e36190a3e6a\" (UID: \"436a09cc-49bc-4784-a4a1-4e36190a3e6a\") " Nov 01 20:27:44.631604 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:27:44.631575 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/436a09cc-49bc-4784-a4a1-4e36190a3e6a-routing-overlay-volume" pod "436a09cc-49bc-4784-a4a1-4e36190a3e6a" (UID: "436a09cc-49bc-4784-a4a1-4e36190a3e6a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:27:44.631671 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:27:44.631589 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/436a09cc-49bc-4784-a4a1-4e36190a3e6a-plugins-config-volume" pod "436a09cc-49bc-4784-a4a1-4e36190a3e6a" (UID: "436a09cc-49bc-4784-a4a1-4e36190a3e6a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:27:44.631842 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:27:44.631800 5935 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/436a09cc-49bc-4784-a4a1-4e36190a3e6a-routing-overlay-volume\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:27:44.631937 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:27:44.631843 5935 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/436a09cc-49bc-4784-a4a1-4e36190a3e6a-plugins-config-volume\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:27:44.633656 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:27:44.633623 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/436a09cc-49bc-4784-a4a1-4e36190a3e6a-kube-api-access-bzvfz" pod "436a09cc-49bc-4784-a4a1-4e36190a3e6a" (UID: "436a09cc-49bc-4784-a4a1-4e36190a3e6a"). InnerVolumeSpecName "kube-api-access-bzvfz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:27:44.633784 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:27:44.633671 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/436a09cc-49bc-4784-a4a1-4e36190a3e6a-provider-credentials" pod "436a09cc-49bc-4784-a4a1-4e36190a3e6a" (UID: "436a09cc-49bc-4784-a4a1-4e36190a3e6a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:27:44.733554 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:27:44.733524 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzvfz\" (UniqueName: \"kubernetes.io/projected/436a09cc-49bc-4784-a4a1-4e36190a3e6a-kube-api-access-bzvfz\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:27:44.733554 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:27:44.733553 5935 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/436a09cc-49bc-4784-a4a1-4e36190a3e6a-provider-credentials\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:27:44.854434 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:27:44.854354 5935 generic.go:356] "Generic (PLEG): container finished" podID="436a09cc-49bc-4784-a4a1-4e36190a3e6a" containerID="24b66071f7a01df256514931a35da71ec034c9390afe00a66ad584dfc9afa139" exitCode=0 Nov 01 20:27:44.854434 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:27:44.854416 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-9ea5d2/payload-processing-external-model-e2e-extmod-b-9ea5d2-86f8rg9hq" Nov 01 20:27:44.854613 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:27:44.854430 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-9ea5d2/payload-processing-external-model-e2e-extmod-b-9ea5d2-86f8rg9hq" event={"ID":"436a09cc-49bc-4784-a4a1-4e36190a3e6a","Type":"ContainerDied","Data":"24b66071f7a01df256514931a35da71ec034c9390afe00a66ad584dfc9afa139"} Nov 01 20:27:44.854613 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:27:44.854507 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-9ea5d2/payload-processing-external-model-e2e-extmod-b-9ea5d2-86f8rg9hq" event={"ID":"436a09cc-49bc-4784-a4a1-4e36190a3e6a","Type":"ContainerDied","Data":"0ae2bb016ee461cd4230d6063d879a7fb78a987e7574267f109aada2b1862abd"} Nov 01 20:27:44.854613 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:27:44.854531 5935 scope.go:122] "RemoveContainer" containerID="24b66071f7a01df256514931a35da71ec034c9390afe00a66ad584dfc9afa139" Nov 01 20:27:44.864520 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:27:44.864501 5935 scope.go:122] "RemoveContainer" containerID="24b66071f7a01df256514931a35da71ec034c9390afe00a66ad584dfc9afa139" Nov 01 20:27:44.864794 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:27:44.864775 5935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24b66071f7a01df256514931a35da71ec034c9390afe00a66ad584dfc9afa139\": container with ID starting with 24b66071f7a01df256514931a35da71ec034c9390afe00a66ad584dfc9afa139 not found: ID does not exist" containerID="24b66071f7a01df256514931a35da71ec034c9390afe00a66ad584dfc9afa139" Nov 01 20:27:44.864871 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:27:44.864802 5935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24b66071f7a01df256514931a35da71ec034c9390afe00a66ad584dfc9afa139"} err="failed to get container status \"24b66071f7a01df256514931a35da71ec034c9390afe00a66ad584dfc9afa139\": rpc error: code = NotFound desc = could not find container \"24b66071f7a01df256514931a35da71ec034c9390afe00a66ad584dfc9afa139\": container with ID starting with 24b66071f7a01df256514931a35da71ec034c9390afe00a66ad584dfc9afa139 not found: ID does not exist" Nov 01 20:27:44.875897 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:27:44.875875 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-9ea5d2/payload-processing-external-model-e2e-extmod-b-9ea5d2-86f8rg9hq"] Nov 01 20:27:44.881088 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:27:44.881065 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-9ea5d2/payload-processing-external-model-e2e-extmod-b-9ea5d2-86f8rg9hq"] Nov 01 20:27:45.416508 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:27:45.416467 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-9ea5d2/payload-processing-external-model-e2e-extmod-b-9ea5d2-86f8rg9hq" podUID="436a09cc-49bc-4784-a4a1-4e36190a3e6a" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.111:50052\" within 1s: context deadline exceeded" Nov 01 20:27:46.353564 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:27:46.353533 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="436a09cc-49bc-4784-a4a1-4e36190a3e6a" path="/var/lib/kubelet/pods/436a09cc-49bc-4784-a4a1-4e36190a3e6a/volumes" Nov 01 20:28:05.844275 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:28:05.844239 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:28:19.445447 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:19.445414 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-9ea5d2-6c96fd4f45-lxrs4"] Nov 01 20:28:19.459534 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:19.459502 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-9ea5d2-78b67f847f-j8dj9"] Nov 01 20:28:24.445864 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:24.445797 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-9ea5d2-6c96fd4f45-lxrs4" podUID="3b4bf215-ad37-4389-b71b-e56de6363784" containerName="payload-processing" containerID="cri-o://49160cc02e2c5108402746b3b2a23508f7ee5c9d001d0c73d6827f34819c1827" gracePeriod=25 Nov 01 20:28:24.460429 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:24.460385 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9ea5d2-78b67f847f-j8dj9" podUID="49ab9daf-af2a-452e-9860-24b64352ca82" containerName="payload-pre-processing" containerID="cri-o://ede909089d65f9ab2435df8571018bf4dcb887ce3a7860c0838744d234505440" gracePeriod=25 Nov 01 20:28:26.395860 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:26.395778 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-9ea5d2-66b589687d-jzxqb"] Nov 01 20:28:26.396360 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:26.396047 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9ea5d2-66b589687d-jzxqb" podUID="52998151-f859-4a20-b075-402dc0d0b6bc" containerName="maas-api" containerID="cri-o://bcc92c5f5755033ce81379889ee4e5f8c7c012600452657fbffa21ef372b04b2" gracePeriod=30 Nov 01 20:28:26.860524 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:26.860501 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9ea5d2-66b589687d-jzxqb" Nov 01 20:28:26.920106 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:26.920076 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52998151-f859-4a20-b075-402dc0d0b6bc-kube-api-access-sfcqc\" (UniqueName: \"kubernetes.io/projected/52998151-f859-4a20-b075-402dc0d0b6bc-kube-api-access-sfcqc\") pod \"52998151-f859-4a20-b075-402dc0d0b6bc\" (UID: \"52998151-f859-4a20-b075-402dc0d0b6bc\") " Nov 01 20:28:26.920277 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:26.920175 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52998151-f859-4a20-b075-402dc0d0b6bc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52998151-f859-4a20-b075-402dc0d0b6bc-maas-api-tls\") pod \"52998151-f859-4a20-b075-402dc0d0b6bc\" (UID: \"52998151-f859-4a20-b075-402dc0d0b6bc\") " Nov 01 20:28:26.920277 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:26.920222 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52998151-f859-4a20-b075-402dc0d0b6bc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/52998151-f859-4a20-b075-402dc0d0b6bc-metrics-certs\") pod \"52998151-f859-4a20-b075-402dc0d0b6bc\" (UID: \"52998151-f859-4a20-b075-402dc0d0b6bc\") " Nov 01 20:28:26.920277 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:26.920272 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52998151-f859-4a20-b075-402dc0d0b6bc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52998151-f859-4a20-b075-402dc0d0b6bc-openshift-service-ca\") pod \"52998151-f859-4a20-b075-402dc0d0b6bc\" (UID: \"52998151-f859-4a20-b075-402dc0d0b6bc\") " Nov 01 20:28:26.920697 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:26.920671 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52998151-f859-4a20-b075-402dc0d0b6bc-openshift-service-ca" pod "52998151-f859-4a20-b075-402dc0d0b6bc" (UID: "52998151-f859-4a20-b075-402dc0d0b6bc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:28:26.920823 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:26.920789 5935 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/52998151-f859-4a20-b075-402dc0d0b6bc-openshift-service-ca\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:28:26.922144 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:26.922118 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52998151-f859-4a20-b075-402dc0d0b6bc-kube-api-access-sfcqc" pod "52998151-f859-4a20-b075-402dc0d0b6bc" (UID: "52998151-f859-4a20-b075-402dc0d0b6bc"). InnerVolumeSpecName "kube-api-access-sfcqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:28:26.922248 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:26.922167 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52998151-f859-4a20-b075-402dc0d0b6bc-metrics-certs" pod "52998151-f859-4a20-b075-402dc0d0b6bc" (UID: "52998151-f859-4a20-b075-402dc0d0b6bc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:28:26.922329 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:26.922254 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52998151-f859-4a20-b075-402dc0d0b6bc-maas-api-tls" pod "52998151-f859-4a20-b075-402dc0d0b6bc" (UID: "52998151-f859-4a20-b075-402dc0d0b6bc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:28:27.022110 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:27.022083 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfcqc\" (UniqueName: \"kubernetes.io/projected/52998151-f859-4a20-b075-402dc0d0b6bc-kube-api-access-sfcqc\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:28:27.022110 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:27.022110 5935 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/52998151-f859-4a20-b075-402dc0d0b6bc-maas-api-tls\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:28:27.022314 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:27.022120 5935 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/52998151-f859-4a20-b075-402dc0d0b6bc-metrics-certs\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:28:27.123369 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:27.123338 5935 generic.go:356] "Generic (PLEG): container finished" podID="52998151-f859-4a20-b075-402dc0d0b6bc" containerID="bcc92c5f5755033ce81379889ee4e5f8c7c012600452657fbffa21ef372b04b2" exitCode=0 Nov 01 20:28:27.123523 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:27.123381 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9ea5d2-66b589687d-jzxqb" Nov 01 20:28:27.123523 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:27.123398 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9ea5d2-66b589687d-jzxqb" event={"ID":"52998151-f859-4a20-b075-402dc0d0b6bc","Type":"ContainerDied","Data":"bcc92c5f5755033ce81379889ee4e5f8c7c012600452657fbffa21ef372b04b2"} Nov 01 20:28:27.123523 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:27.123438 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-9ea5d2-66b589687d-jzxqb" event={"ID":"52998151-f859-4a20-b075-402dc0d0b6bc","Type":"ContainerDied","Data":"38d3e7d120dc6554ed38d78c22d783251829ed78c9ac791ba3d0674f5a0baab2"} Nov 01 20:28:27.123523 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:27.123459 5935 scope.go:122] "RemoveContainer" containerID="bcc92c5f5755033ce81379889ee4e5f8c7c012600452657fbffa21ef372b04b2" Nov 01 20:28:27.133705 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:27.133688 5935 scope.go:122] "RemoveContainer" containerID="bcc92c5f5755033ce81379889ee4e5f8c7c012600452657fbffa21ef372b04b2" Nov 01 20:28:27.133969 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:28:27.133952 5935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcc92c5f5755033ce81379889ee4e5f8c7c012600452657fbffa21ef372b04b2\": container with ID starting with bcc92c5f5755033ce81379889ee4e5f8c7c012600452657fbffa21ef372b04b2 not found: ID does not exist" containerID="bcc92c5f5755033ce81379889ee4e5f8c7c012600452657fbffa21ef372b04b2" Nov 01 20:28:27.134041 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:27.133974 5935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcc92c5f5755033ce81379889ee4e5f8c7c012600452657fbffa21ef372b04b2"} err="failed to get container status \"bcc92c5f5755033ce81379889ee4e5f8c7c012600452657fbffa21ef372b04b2\": rpc error: code = NotFound desc = could not find container \"bcc92c5f5755033ce81379889ee4e5f8c7c012600452657fbffa21ef372b04b2\": container with ID starting with bcc92c5f5755033ce81379889ee4e5f8c7c012600452657fbffa21ef372b04b2 not found: ID does not exist" Nov 01 20:28:27.145167 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:27.145149 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-9ea5d2-66b589687d-jzxqb"] Nov 01 20:28:27.153444 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:27.153417 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-9ea5d2-66b589687d-jzxqb"] Nov 01 20:28:27.574611 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:27.574570 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-9ea5d2-6c96fd4f45-lxrs4" podUID="3b4bf215-ad37-4389-b71b-e56de6363784" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 20:28:27.574999 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:27.574640 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9ea5d2-78b67f847f-j8dj9" podUID="49ab9daf-af2a-452e-9860-24b64352ca82" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 20:28:28.354432 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:28.354400 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52998151-f859-4a20-b075-402dc0d0b6bc" path="/var/lib/kubelet/pods/52998151-f859-4a20-b075-402dc0d0b6bc/volumes" Nov 01 20:28:32.575358 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:32.575315 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p"] Nov 01 20:28:33.165943 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:33.165893 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p" podUID="0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0" containerName="istio-proxy" containerID="cri-o://2e67ceba65fcea7b37dd46674360ec26ea655750622f4ebc21bbac674303a3bf" gracePeriod=30 Nov 01 20:28:35.843577 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:28:35.843541 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:28:37.574556 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:37.574511 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-9ea5d2-6c96fd4f45-lxrs4" podUID="3b4bf215-ad37-4389-b71b-e56de6363784" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 20:28:37.574556 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:37.574529 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9ea5d2-78b67f847f-j8dj9" podUID="49ab9daf-af2a-452e-9860-24b64352ca82" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 20:28:38.433464 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:38.433409 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p" Nov 01 20:28:38.472401 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:38.472376 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0-istiod-ca-cert\") pod \"0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0\" (UID: \"0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0\") " Nov 01 20:28:38.472551 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:38.472413 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0-workload-certs\") pod \"0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0\" (UID: \"0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0\") " Nov 01 20:28:38.472551 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:38.472442 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0-istio-token\" (UniqueName: \"kubernetes.io/projected/0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0-istio-token\") pod \"0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0\" (UID: \"0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0\") " Nov 01 20:28:38.472551 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:38.472482 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0-istio-podinfo\") pod \"0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0\" (UID: \"0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0\") " Nov 01 20:28:38.472551 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:38.472521 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0-istio-data\") pod \"0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0\" (UID: \"0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0\") " Nov 01 20:28:38.472773 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:38.472568 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0-kube-api-access-k9cpb\" (UniqueName: \"kubernetes.io/projected/0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0-kube-api-access-k9cpb\") pod \"0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0\" (UID: \"0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0\") " Nov 01 20:28:38.472773 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:38.472688 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0-credential-socket\") pod \"0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0\" (UID: \"0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0\") " Nov 01 20:28:38.472773 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:38.472673 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0-workload-certs" pod "0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0" (UID: "0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:28:38.472773 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:38.472741 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0-workload-socket\") pod \"0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0\" (UID: \"0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0\") " Nov 01 20:28:38.472986 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:38.472798 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0-istio-envoy\") pod \"0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0\" (UID: \"0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0\") " Nov 01 20:28:38.472986 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:38.472857 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0-istio-data" pod "0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0" (UID: "0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:28:38.472986 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:38.472859 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0-istiod-ca-cert" pod "0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0" (UID: "0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:28:38.473338 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:38.473269 5935 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0-istio-data\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:28:38.473338 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:38.473308 5935 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0-istiod-ca-cert\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:28:38.473338 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:38.473322 5935 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0-workload-certs\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:28:38.473549 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:38.473055 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0-credential-socket" pod "0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0" (UID: "0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:28:38.473549 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:38.473113 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0-workload-socket" pod "0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0" (UID: "0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:28:38.475086 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:38.475059 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0-istio-podinfo" pod "0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0" (UID: "0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:28:38.475235 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:38.475214 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0-istio-envoy" pod "0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0" (UID: "0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:28:38.475339 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:38.475320 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0-istio-token" pod "0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0" (UID: "0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:28:38.475405 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:38.475360 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0-kube-api-access-k9cpb" pod "0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0" (UID: "0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0"). InnerVolumeSpecName "kube-api-access-k9cpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:28:38.574916 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:38.574889 5935 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0-credential-socket\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:28:38.574916 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:38.574915 5935 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0-workload-socket\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:28:38.575306 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:38.574925 5935 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0-istio-envoy\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:28:38.575306 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:38.574934 5935 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0-istio-token\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:28:38.575306 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:38.574942 5935 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0-istio-podinfo\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:28:38.575306 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:38.574951 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9cpb\" (UniqueName: \"kubernetes.io/projected/0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0-kube-api-access-k9cpb\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:28:39.204804 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:39.204767 5935 generic.go:356] "Generic (PLEG): container finished" podID="0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0" containerID="2e67ceba65fcea7b37dd46674360ec26ea655750622f4ebc21bbac674303a3bf" exitCode=0 Nov 01 20:28:39.205007 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:39.204822 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p" Nov 01 20:28:39.205007 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:39.204855 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p" event={"ID":"0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0","Type":"ContainerDied","Data":"2e67ceba65fcea7b37dd46674360ec26ea655750622f4ebc21bbac674303a3bf"} Nov 01 20:28:39.205007 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:39.204922 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p" event={"ID":"0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0","Type":"ContainerDied","Data":"8f0ee3006d566e49cba910528569f8baab86691911006f75c1da310a5ef015e8"} Nov 01 20:28:39.205007 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:39.204953 5935 scope.go:122] "RemoveContainer" containerID="2e67ceba65fcea7b37dd46674360ec26ea655750622f4ebc21bbac674303a3bf" Nov 01 20:28:39.215773 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:39.215758 5935 scope.go:122] "RemoveContainer" containerID="9e0b2bf5d91d0d6af1a2f8176065780be9ccb24b68e00573db4a0ffb1c2c086a" Nov 01 20:28:39.225928 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:39.225909 5935 scope.go:122] "RemoveContainer" containerID="2e67ceba65fcea7b37dd46674360ec26ea655750622f4ebc21bbac674303a3bf" Nov 01 20:28:39.226199 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:28:39.226180 5935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e67ceba65fcea7b37dd46674360ec26ea655750622f4ebc21bbac674303a3bf\": container with ID starting with 2e67ceba65fcea7b37dd46674360ec26ea655750622f4ebc21bbac674303a3bf not found: ID does not exist" containerID="2e67ceba65fcea7b37dd46674360ec26ea655750622f4ebc21bbac674303a3bf" Nov 01 20:28:39.226318 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:39.226212 5935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e67ceba65fcea7b37dd46674360ec26ea655750622f4ebc21bbac674303a3bf"} err="failed to get container status \"2e67ceba65fcea7b37dd46674360ec26ea655750622f4ebc21bbac674303a3bf\": rpc error: code = NotFound desc = could not find container \"2e67ceba65fcea7b37dd46674360ec26ea655750622f4ebc21bbac674303a3bf\": container with ID starting with 2e67ceba65fcea7b37dd46674360ec26ea655750622f4ebc21bbac674303a3bf not found: ID does not exist" Nov 01 20:28:39.226318 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:39.226237 5935 scope.go:122] "RemoveContainer" containerID="9e0b2bf5d91d0d6af1a2f8176065780be9ccb24b68e00573db4a0ffb1c2c086a" Nov 01 20:28:39.226586 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:28:39.226569 5935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e0b2bf5d91d0d6af1a2f8176065780be9ccb24b68e00573db4a0ffb1c2c086a\": container with ID starting with 9e0b2bf5d91d0d6af1a2f8176065780be9ccb24b68e00573db4a0ffb1c2c086a not found: ID does not exist" containerID="9e0b2bf5d91d0d6af1a2f8176065780be9ccb24b68e00573db4a0ffb1c2c086a" Nov 01 20:28:39.226652 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:39.226594 5935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e0b2bf5d91d0d6af1a2f8176065780be9ccb24b68e00573db4a0ffb1c2c086a"} err="failed to get container status \"9e0b2bf5d91d0d6af1a2f8176065780be9ccb24b68e00573db4a0ffb1c2c086a\": rpc error: code = NotFound desc = could not find container \"9e0b2bf5d91d0d6af1a2f8176065780be9ccb24b68e00573db4a0ffb1c2c086a\": container with ID starting with 9e0b2bf5d91d0d6af1a2f8176065780be9ccb24b68e00573db4a0ffb1c2c086a not found: ID does not exist" Nov 01 20:28:39.227896 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:39.227878 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p"] Nov 01 20:28:39.232880 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:39.232860 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-9ea5d2-openshift-default-5cd6fcfc77-cqw4p"] Nov 01 20:28:40.353113 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:40.353084 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0" path="/var/lib/kubelet/pods/0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0/volumes" Nov 01 20:28:42.605684 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:42.605646 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d2dc0ad5-ff87549c8-xs57b"] Nov 01 20:28:42.606527 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:42.606502 5935 cpu_manager.go:414] "containerMap: removing container" podUID="0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0" containerName="istio-proxy" Nov 01 20:28:42.606639 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:42.606528 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0" containerName="istio-proxy" Nov 01 20:28:42.606639 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:42.606556 5935 cpu_manager.go:414] "containerMap: removing container" podUID="52998151-f859-4a20-b075-402dc0d0b6bc" containerName="maas-api" Nov 01 20:28:42.606639 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:42.606564 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52998151-f859-4a20-b075-402dc0d0b6bc" containerName="maas-api" Nov 01 20:28:42.606639 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:42.606587 5935 cpu_manager.go:414] "containerMap: removing container" podUID="0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0" containerName="istio-proxy" Nov 01 20:28:42.606639 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:42.606595 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0" containerName="istio-proxy" Nov 01 20:28:42.606639 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:42.606618 5935 cpu_manager.go:414] "containerMap: removing container" podUID="8686280e-385b-4a98-a1d2-5018e5f08fc1" containerName="istio-proxy" Nov 01 20:28:42.606639 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:42.606625 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8686280e-385b-4a98-a1d2-5018e5f08fc1" containerName="istio-proxy" Nov 01 20:28:42.607002 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:42.606649 5935 cpu_manager.go:414] "containerMap: removing container" podUID="436a09cc-49bc-4784-a4a1-4e36190a3e6a" containerName="payload-processing" Nov 01 20:28:42.607002 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:42.606657 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="436a09cc-49bc-4784-a4a1-4e36190a3e6a" containerName="payload-processing" Nov 01 20:28:42.607002 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:42.606777 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="8686280e-385b-4a98-a1d2-5018e5f08fc1" containerName="istio-proxy" Nov 01 20:28:42.607002 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:42.606793 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="436a09cc-49bc-4784-a4a1-4e36190a3e6a" containerName="payload-processing" Nov 01 20:28:42.607002 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:42.606810 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0" containerName="istio-proxy" Nov 01 20:28:42.607002 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:42.606822 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="52998151-f859-4a20-b075-402dc0d0b6bc" containerName="maas-api" Nov 01 20:28:42.607002 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:42.606832 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a36a3c4-57f6-4486-9adb-0ef1af2fb3d0" containerName="istio-proxy" Nov 01 20:28:42.612261 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:42.612237 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d2dc0ad5-ff87549c8-xs57b" Nov 01 20:28:42.618085 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:42.618060 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d2dc0ad5-ff87549c8-xs57b"] Nov 01 20:28:42.621498 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:42.621476 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-d2dc0ad5-serving-cert\"" Nov 01 20:28:42.723668 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:42.723637 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cdc5a94-7c68-49f2-95b9-8d23d1355581-maas-api-tls\") pod \"maas-api-e2e-mt-d2dc0ad5-ff87549c8-xs57b\" (UID: \"7cdc5a94-7c68-49f2-95b9-8d23d1355581\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d2dc0ad5-ff87549c8-xs57b" Nov 01 20:28:42.723833 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:42.723697 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cdc5a94-7c68-49f2-95b9-8d23d1355581-openshift-service-ca\") pod \"maas-api-e2e-mt-d2dc0ad5-ff87549c8-xs57b\" (UID: \"7cdc5a94-7c68-49f2-95b9-8d23d1355581\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d2dc0ad5-ff87549c8-xs57b" Nov 01 20:28:42.723833 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:42.723808 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czkq2\" (UniqueName: \"kubernetes.io/projected/7cdc5a94-7c68-49f2-95b9-8d23d1355581-kube-api-access-czkq2\") pod \"maas-api-e2e-mt-d2dc0ad5-ff87549c8-xs57b\" (UID: \"7cdc5a94-7c68-49f2-95b9-8d23d1355581\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d2dc0ad5-ff87549c8-xs57b" Nov 01 20:28:42.723916 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:42.723868 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cdc5a94-7c68-49f2-95b9-8d23d1355581-metrics-certs\") pod \"maas-api-e2e-mt-d2dc0ad5-ff87549c8-xs57b\" (UID: \"7cdc5a94-7c68-49f2-95b9-8d23d1355581\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d2dc0ad5-ff87549c8-xs57b" Nov 01 20:28:42.825626 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:42.825585 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cdc5a94-7c68-49f2-95b9-8d23d1355581-maas-api-tls\") pod \"maas-api-e2e-mt-d2dc0ad5-ff87549c8-xs57b\" (UID: \"7cdc5a94-7c68-49f2-95b9-8d23d1355581\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d2dc0ad5-ff87549c8-xs57b" Nov 01 20:28:42.825838 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:42.825652 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cdc5a94-7c68-49f2-95b9-8d23d1355581-openshift-service-ca\") pod \"maas-api-e2e-mt-d2dc0ad5-ff87549c8-xs57b\" (UID: \"7cdc5a94-7c68-49f2-95b9-8d23d1355581\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d2dc0ad5-ff87549c8-xs57b" Nov 01 20:28:42.825838 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:42.825689 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czkq2\" (UniqueName: \"kubernetes.io/projected/7cdc5a94-7c68-49f2-95b9-8d23d1355581-kube-api-access-czkq2\") pod \"maas-api-e2e-mt-d2dc0ad5-ff87549c8-xs57b\" (UID: \"7cdc5a94-7c68-49f2-95b9-8d23d1355581\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d2dc0ad5-ff87549c8-xs57b" Nov 01 20:28:42.825838 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:42.825719 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cdc5a94-7c68-49f2-95b9-8d23d1355581-metrics-certs\") pod \"maas-api-e2e-mt-d2dc0ad5-ff87549c8-xs57b\" (UID: \"7cdc5a94-7c68-49f2-95b9-8d23d1355581\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d2dc0ad5-ff87549c8-xs57b" Nov 01 20:28:42.826574 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:42.826542 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cdc5a94-7c68-49f2-95b9-8d23d1355581-openshift-service-ca\") pod \"maas-api-e2e-mt-d2dc0ad5-ff87549c8-xs57b\" (UID: \"7cdc5a94-7c68-49f2-95b9-8d23d1355581\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d2dc0ad5-ff87549c8-xs57b" Nov 01 20:28:42.828607 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:42.828580 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cdc5a94-7c68-49f2-95b9-8d23d1355581-metrics-certs\") pod \"maas-api-e2e-mt-d2dc0ad5-ff87549c8-xs57b\" (UID: \"7cdc5a94-7c68-49f2-95b9-8d23d1355581\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d2dc0ad5-ff87549c8-xs57b" Nov 01 20:28:42.828706 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:42.828635 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cdc5a94-7c68-49f2-95b9-8d23d1355581-maas-api-tls\") pod \"maas-api-e2e-mt-d2dc0ad5-ff87549c8-xs57b\" (UID: \"7cdc5a94-7c68-49f2-95b9-8d23d1355581\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d2dc0ad5-ff87549c8-xs57b" Nov 01 20:28:42.833726 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:42.833697 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czkq2\" (UniqueName: \"kubernetes.io/projected/7cdc5a94-7c68-49f2-95b9-8d23d1355581-kube-api-access-czkq2\") pod \"maas-api-e2e-mt-d2dc0ad5-ff87549c8-xs57b\" (UID: \"7cdc5a94-7c68-49f2-95b9-8d23d1355581\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-d2dc0ad5-ff87549c8-xs57b" Nov 01 20:28:42.933809 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:42.933736 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d2dc0ad5-ff87549c8-xs57b" Nov 01 20:28:43.214881 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:43.214855 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d2dc0ad5-ff87549c8-xs57b"] Nov 01 20:28:43.216835 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:28:43.216811 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cdc5a94_7c68_49f2_95b9_8d23d1355581.slice/crio-3ddf4426a0bdf9657bebb48c021481e190f7dacb5f7eddfefa7462df33e2bee8 WatchSource:0}: Error finding container 3ddf4426a0bdf9657bebb48c021481e190f7dacb5f7eddfefa7462df33e2bee8: Status 404 returned error can't find the container with id 3ddf4426a0bdf9657bebb48c021481e190f7dacb5f7eddfefa7462df33e2bee8 Nov 01 20:28:43.218898 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:43.218876 5935 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 20:28:43.238631 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:43.238598 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d2dc0ad5-ff87549c8-xs57b" event={"ID":"7cdc5a94-7c68-49f2-95b9-8d23d1355581","Type":"ContainerStarted","Data":"3ddf4426a0bdf9657bebb48c021481e190f7dacb5f7eddfefa7462df33e2bee8"} Nov 01 20:28:44.245885 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:44.245848 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d2dc0ad5-ff87549c8-xs57b" event={"ID":"7cdc5a94-7c68-49f2-95b9-8d23d1355581","Type":"ContainerStarted","Data":"ca9fede5f97956aa16b8ff2e5d5f554da676983120099d42db6d91dc25846b5f"} Nov 01 20:28:44.246303 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:44.246009 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d2dc0ad5-ff87549c8-xs57b" Nov 01 20:28:44.263982 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:44.263921 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d2dc0ad5-ff87549c8-xs57b" podStartSLOduration=1.7025329390000001 podStartE2EDuration="2.263907375s" totalImagesPullingTime="561.374436ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:28:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:28:44.26254604 +0000 UTC m=+1770.035706573" watchObservedRunningTime="2026-11-01 20:28:44.263907375 +0000 UTC m=+1770.037067889" Nov 01 20:28:44.728734 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:44.728707 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-9ea5d2-6c96fd4f45-lxrs4" Nov 01 20:28:44.740738 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:44.740712 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9ea5d2-78b67f847f-j8dj9" Nov 01 20:28:44.851239 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:44.851211 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b4bf215-ad37-4389-b71b-e56de6363784-kube-api-access-794xj\" (UniqueName: \"kubernetes.io/projected/3b4bf215-ad37-4389-b71b-e56de6363784-kube-api-access-794xj\") pod \"3b4bf215-ad37-4389-b71b-e56de6363784\" (UID: \"3b4bf215-ad37-4389-b71b-e56de6363784\") " Nov 01 20:28:44.851417 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:44.851259 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49ab9daf-af2a-452e-9860-24b64352ca82-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49ab9daf-af2a-452e-9860-24b64352ca82-plugins-config-volume\") pod \"49ab9daf-af2a-452e-9860-24b64352ca82\" (UID: \"49ab9daf-af2a-452e-9860-24b64352ca82\") " Nov 01 20:28:44.851417 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:44.851372 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49ab9daf-af2a-452e-9860-24b64352ca82-kube-api-access-gvd6z\" (UniqueName: \"kubernetes.io/projected/49ab9daf-af2a-452e-9860-24b64352ca82-kube-api-access-gvd6z\") pod \"49ab9daf-af2a-452e-9860-24b64352ca82\" (UID: \"49ab9daf-af2a-452e-9860-24b64352ca82\") " Nov 01 20:28:44.851534 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:44.851519 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b4bf215-ad37-4389-b71b-e56de6363784-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b4bf215-ad37-4389-b71b-e56de6363784-plugins-config-volume\") pod \"3b4bf215-ad37-4389-b71b-e56de6363784\" (UID: \"3b4bf215-ad37-4389-b71b-e56de6363784\") " Nov 01 20:28:44.852090 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:44.852064 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49ab9daf-af2a-452e-9860-24b64352ca82-plugins-config-volume" pod "49ab9daf-af2a-452e-9860-24b64352ca82" (UID: "49ab9daf-af2a-452e-9860-24b64352ca82"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:28:44.852700 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:44.852674 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b4bf215-ad37-4389-b71b-e56de6363784-plugins-config-volume" pod "3b4bf215-ad37-4389-b71b-e56de6363784" (UID: "3b4bf215-ad37-4389-b71b-e56de6363784"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:28:44.853947 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:44.853921 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b4bf215-ad37-4389-b71b-e56de6363784-kube-api-access-794xj" pod "3b4bf215-ad37-4389-b71b-e56de6363784" (UID: "3b4bf215-ad37-4389-b71b-e56de6363784"). InnerVolumeSpecName "kube-api-access-794xj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:28:44.855098 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:44.855077 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49ab9daf-af2a-452e-9860-24b64352ca82-kube-api-access-gvd6z" pod "49ab9daf-af2a-452e-9860-24b64352ca82" (UID: "49ab9daf-af2a-452e-9860-24b64352ca82"). InnerVolumeSpecName "kube-api-access-gvd6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:28:44.953343 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:44.953310 5935 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b4bf215-ad37-4389-b71b-e56de6363784-plugins-config-volume\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:28:44.953343 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:44.953341 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-794xj\" (UniqueName: \"kubernetes.io/projected/3b4bf215-ad37-4389-b71b-e56de6363784-kube-api-access-794xj\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:28:44.953343 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:44.953351 5935 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49ab9daf-af2a-452e-9860-24b64352ca82-plugins-config-volume\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:28:44.953629 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:44.953360 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvd6z\" (UniqueName: \"kubernetes.io/projected/49ab9daf-af2a-452e-9860-24b64352ca82-kube-api-access-gvd6z\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:28:45.252360 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:45.252327 5935 generic.go:356] "Generic (PLEG): container finished" podID="49ab9daf-af2a-452e-9860-24b64352ca82" containerID="ede909089d65f9ab2435df8571018bf4dcb887ce3a7860c0838744d234505440" exitCode=0 Nov 01 20:28:45.252858 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:45.252370 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9ea5d2-78b67f847f-j8dj9" Nov 01 20:28:45.252858 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:45.252418 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9ea5d2-78b67f847f-j8dj9" event={"ID":"49ab9daf-af2a-452e-9860-24b64352ca82","Type":"ContainerDied","Data":"ede909089d65f9ab2435df8571018bf4dcb887ce3a7860c0838744d234505440"} Nov 01 20:28:45.252858 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:45.252481 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9ea5d2-78b67f847f-j8dj9" event={"ID":"49ab9daf-af2a-452e-9860-24b64352ca82","Type":"ContainerDied","Data":"e7113e6207259d97602348d4dd988c8d6a75a30a7b6010b89c4c5300586a1d5d"} Nov 01 20:28:45.252858 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:45.252517 5935 scope.go:122] "RemoveContainer" containerID="ede909089d65f9ab2435df8571018bf4dcb887ce3a7860c0838744d234505440" Nov 01 20:28:45.253792 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:45.253774 5935 generic.go:356] "Generic (PLEG): container finished" podID="3b4bf215-ad37-4389-b71b-e56de6363784" containerID="49160cc02e2c5108402746b3b2a23508f7ee5c9d001d0c73d6827f34819c1827" exitCode=0 Nov 01 20:28:45.253884 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:45.253823 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-9ea5d2-6c96fd4f45-lxrs4" Nov 01 20:28:45.253884 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:45.253850 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-9ea5d2-6c96fd4f45-lxrs4" event={"ID":"3b4bf215-ad37-4389-b71b-e56de6363784","Type":"ContainerDied","Data":"49160cc02e2c5108402746b3b2a23508f7ee5c9d001d0c73d6827f34819c1827"} Nov 01 20:28:45.254008 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:45.253895 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-9ea5d2-6c96fd4f45-lxrs4" event={"ID":"3b4bf215-ad37-4389-b71b-e56de6363784","Type":"ContainerDied","Data":"0d9b4fe1473d6e4695c71667e2eb98f6bae84d0ce079a2b6f140ab40fb1bb384"} Nov 01 20:28:45.264198 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:45.264181 5935 scope.go:122] "RemoveContainer" containerID="ede909089d65f9ab2435df8571018bf4dcb887ce3a7860c0838744d234505440" Nov 01 20:28:45.264503 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:28:45.264471 5935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ede909089d65f9ab2435df8571018bf4dcb887ce3a7860c0838744d234505440\": container with ID starting with ede909089d65f9ab2435df8571018bf4dcb887ce3a7860c0838744d234505440 not found: ID does not exist" containerID="ede909089d65f9ab2435df8571018bf4dcb887ce3a7860c0838744d234505440" Nov 01 20:28:45.264503 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:45.264493 5935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ede909089d65f9ab2435df8571018bf4dcb887ce3a7860c0838744d234505440"} err="failed to get container status \"ede909089d65f9ab2435df8571018bf4dcb887ce3a7860c0838744d234505440\": rpc error: code = NotFound desc = could not find container \"ede909089d65f9ab2435df8571018bf4dcb887ce3a7860c0838744d234505440\": container with ID starting with ede909089d65f9ab2435df8571018bf4dcb887ce3a7860c0838744d234505440 not found: ID does not exist" Nov 01 20:28:45.264637 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:45.264509 5935 scope.go:122] "RemoveContainer" containerID="49160cc02e2c5108402746b3b2a23508f7ee5c9d001d0c73d6827f34819c1827" Nov 01 20:28:45.274316 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:45.274179 5935 scope.go:122] "RemoveContainer" containerID="49160cc02e2c5108402746b3b2a23508f7ee5c9d001d0c73d6827f34819c1827" Nov 01 20:28:45.274574 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:28:45.274546 5935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49160cc02e2c5108402746b3b2a23508f7ee5c9d001d0c73d6827f34819c1827\": container with ID starting with 49160cc02e2c5108402746b3b2a23508f7ee5c9d001d0c73d6827f34819c1827 not found: ID does not exist" containerID="49160cc02e2c5108402746b3b2a23508f7ee5c9d001d0c73d6827f34819c1827" Nov 01 20:28:45.274666 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:45.274582 5935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49160cc02e2c5108402746b3b2a23508f7ee5c9d001d0c73d6827f34819c1827"} err="failed to get container status \"49160cc02e2c5108402746b3b2a23508f7ee5c9d001d0c73d6827f34819c1827\": rpc error: code = NotFound desc = could not find container \"49160cc02e2c5108402746b3b2a23508f7ee5c9d001d0c73d6827f34819c1827\": container with ID starting with 49160cc02e2c5108402746b3b2a23508f7ee5c9d001d0c73d6827f34819c1827 not found: ID does not exist" Nov 01 20:28:45.287493 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:45.287454 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-9ea5d2-6c96fd4f45-lxrs4"] Nov 01 20:28:45.293490 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:45.293433 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-9ea5d2-6c96fd4f45-lxrs4"] Nov 01 20:28:45.323317 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:45.318402 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-9ea5d2-78b67f847f-j8dj9"] Nov 01 20:28:45.323317 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:45.322088 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-9ea5d2-78b67f847f-j8dj9"] Nov 01 20:28:46.356610 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:46.356582 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b4bf215-ad37-4389-b71b-e56de6363784" path="/var/lib/kubelet/pods/3b4bf215-ad37-4389-b71b-e56de6363784/volumes" Nov 01 20:28:46.357087 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:46.357029 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49ab9daf-af2a-452e-9860-24b64352ca82" path="/var/lib/kubelet/pods/49ab9daf-af2a-452e-9860-24b64352ca82/volumes" Nov 01 20:28:50.259840 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:28:50.259807 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d2dc0ad5-ff87549c8-xs57b" Nov 01 20:29:05.361261 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:05.361222 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr"] Nov 01 20:29:05.363134 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:05.363107 5935 cpu_manager.go:414] "containerMap: removing container" podUID="49ab9daf-af2a-452e-9860-24b64352ca82" containerName="payload-pre-processing" Nov 01 20:29:05.363326 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:05.363310 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49ab9daf-af2a-452e-9860-24b64352ca82" containerName="payload-pre-processing" Nov 01 20:29:05.363502 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:05.363487 5935 cpu_manager.go:414] "containerMap: removing container" podUID="3b4bf215-ad37-4389-b71b-e56de6363784" containerName="payload-processing" Nov 01 20:29:05.363590 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:05.363579 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b4bf215-ad37-4389-b71b-e56de6363784" containerName="payload-processing" Nov 01 20:29:05.363831 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:05.363816 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="49ab9daf-af2a-452e-9860-24b64352ca82" containerName="payload-pre-processing" Nov 01 20:29:05.363945 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:05.363935 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b4bf215-ad37-4389-b71b-e56de6363784" containerName="payload-processing" Nov 01 20:29:05.369419 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:05.369361 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr" Nov 01 20:29:05.373382 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:05.373363 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-8b6cd6-openshift-default-dockercfg-tnb8t\"" Nov 01 20:29:05.382071 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:05.382048 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr"] Nov 01 20:29:05.430436 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:05.430401 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92a72792-1735-4970-88ac-ca2a50a5b63e-istio-podinfo\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr\" (UID: \"92a72792-1735-4970-88ac-ca2a50a5b63e\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr" Nov 01 20:29:05.430624 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:05.430445 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92a72792-1735-4970-88ac-ca2a50a5b63e-istio-data\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr\" (UID: \"92a72792-1735-4970-88ac-ca2a50a5b63e\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr" Nov 01 20:29:05.430624 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:05.430541 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92a72792-1735-4970-88ac-ca2a50a5b63e-istio-token\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr\" (UID: \"92a72792-1735-4970-88ac-ca2a50a5b63e\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr" Nov 01 20:29:05.430743 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:05.430681 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92a72792-1735-4970-88ac-ca2a50a5b63e-istio-envoy\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr\" (UID: \"92a72792-1735-4970-88ac-ca2a50a5b63e\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr" Nov 01 20:29:05.430743 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:05.430712 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92a72792-1735-4970-88ac-ca2a50a5b63e-workload-certs\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr\" (UID: \"92a72792-1735-4970-88ac-ca2a50a5b63e\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr" Nov 01 20:29:05.430846 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:05.430767 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a72792-1735-4970-88ac-ca2a50a5b63e-credential-socket\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr\" (UID: \"92a72792-1735-4970-88ac-ca2a50a5b63e\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr" Nov 01 20:29:05.430921 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:05.430898 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgfxc\" (UniqueName: \"kubernetes.io/projected/92a72792-1735-4970-88ac-ca2a50a5b63e-kube-api-access-vgfxc\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr\" (UID: \"92a72792-1735-4970-88ac-ca2a50a5b63e\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr" Nov 01 20:29:05.430977 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:05.430953 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a72792-1735-4970-88ac-ca2a50a5b63e-workload-socket\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr\" (UID: \"92a72792-1735-4970-88ac-ca2a50a5b63e\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr" Nov 01 20:29:05.431035 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:05.430982 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92a72792-1735-4970-88ac-ca2a50a5b63e-istiod-ca-cert\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr\" (UID: \"92a72792-1735-4970-88ac-ca2a50a5b63e\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr" Nov 01 20:29:05.532103 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:05.532065 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgfxc\" (UniqueName: \"kubernetes.io/projected/92a72792-1735-4970-88ac-ca2a50a5b63e-kube-api-access-vgfxc\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr\" (UID: \"92a72792-1735-4970-88ac-ca2a50a5b63e\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr" Nov 01 20:29:05.532328 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:05.532122 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a72792-1735-4970-88ac-ca2a50a5b63e-workload-socket\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr\" (UID: \"92a72792-1735-4970-88ac-ca2a50a5b63e\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr" Nov 01 20:29:05.532328 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:05.532152 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92a72792-1735-4970-88ac-ca2a50a5b63e-istiod-ca-cert\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr\" (UID: \"92a72792-1735-4970-88ac-ca2a50a5b63e\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr" Nov 01 20:29:05.532328 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:05.532186 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92a72792-1735-4970-88ac-ca2a50a5b63e-istio-podinfo\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr\" (UID: \"92a72792-1735-4970-88ac-ca2a50a5b63e\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr" Nov 01 20:29:05.532328 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:05.532226 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92a72792-1735-4970-88ac-ca2a50a5b63e-istio-data\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr\" (UID: \"92a72792-1735-4970-88ac-ca2a50a5b63e\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr" Nov 01 20:29:05.532328 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:05.532278 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92a72792-1735-4970-88ac-ca2a50a5b63e-istio-token\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr\" (UID: \"92a72792-1735-4970-88ac-ca2a50a5b63e\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr" Nov 01 20:29:05.532618 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:05.532498 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92a72792-1735-4970-88ac-ca2a50a5b63e-istio-envoy\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr\" (UID: \"92a72792-1735-4970-88ac-ca2a50a5b63e\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr" Nov 01 20:29:05.532618 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:05.532531 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92a72792-1735-4970-88ac-ca2a50a5b63e-workload-certs\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr\" (UID: \"92a72792-1735-4970-88ac-ca2a50a5b63e\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr" Nov 01 20:29:05.532618 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:05.532608 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a72792-1735-4970-88ac-ca2a50a5b63e-credential-socket\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr\" (UID: \"92a72792-1735-4970-88ac-ca2a50a5b63e\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr" Nov 01 20:29:05.532787 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:05.532640 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a72792-1735-4970-88ac-ca2a50a5b63e-workload-socket\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr\" (UID: \"92a72792-1735-4970-88ac-ca2a50a5b63e\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr" Nov 01 20:29:05.532787 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:05.532702 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92a72792-1735-4970-88ac-ca2a50a5b63e-istio-data\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr\" (UID: \"92a72792-1735-4970-88ac-ca2a50a5b63e\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr" Nov 01 20:29:05.532957 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:05.532933 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92a72792-1735-4970-88ac-ca2a50a5b63e-workload-certs\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr\" (UID: \"92a72792-1735-4970-88ac-ca2a50a5b63e\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr" Nov 01 20:29:05.533102 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:05.533081 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a72792-1735-4970-88ac-ca2a50a5b63e-credential-socket\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr\" (UID: \"92a72792-1735-4970-88ac-ca2a50a5b63e\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr" Nov 01 20:29:05.533211 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:05.533165 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92a72792-1735-4970-88ac-ca2a50a5b63e-istiod-ca-cert\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr\" (UID: \"92a72792-1735-4970-88ac-ca2a50a5b63e\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr" Nov 01 20:29:05.534872 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:05.534850 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92a72792-1735-4970-88ac-ca2a50a5b63e-istio-envoy\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr\" (UID: \"92a72792-1735-4970-88ac-ca2a50a5b63e\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr" Nov 01 20:29:05.534957 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:05.534857 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92a72792-1735-4970-88ac-ca2a50a5b63e-istio-podinfo\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr\" (UID: \"92a72792-1735-4970-88ac-ca2a50a5b63e\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr" Nov 01 20:29:05.539592 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:05.539566 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92a72792-1735-4970-88ac-ca2a50a5b63e-istio-token\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr\" (UID: \"92a72792-1735-4970-88ac-ca2a50a5b63e\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr" Nov 01 20:29:05.539592 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:05.539576 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgfxc\" (UniqueName: \"kubernetes.io/projected/92a72792-1735-4970-88ac-ca2a50a5b63e-kube-api-access-vgfxc\") pod \"e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr\" (UID: \"92a72792-1735-4970-88ac-ca2a50a5b63e\") " pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr" Nov 01 20:29:05.688547 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:05.688479 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr" Nov 01 20:29:05.844846 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:29:05.844808 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:29:05.971919 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:05.971892 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr"] Nov 01 20:29:05.973171 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:29:05.973145 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92a72792_1735_4970_88ac_ca2a50a5b63e.slice/crio-fe104f2f2528f08751fa1f4668202b15803a934d51fe4b18368a9220111a35b7 WatchSource:0}: Error finding container fe104f2f2528f08751fa1f4668202b15803a934d51fe4b18368a9220111a35b7: Status 404 returned error can't find the container with id fe104f2f2528f08751fa1f4668202b15803a934d51fe4b18368a9220111a35b7 Nov 01 20:29:05.975925 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:05.975883 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:29:05.975993 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:05.975974 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:29:05.976066 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:05.976050 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:29:06.439047 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:06.438963 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr" event={"ID":"92a72792-1735-4970-88ac-ca2a50a5b63e","Type":"ContainerStarted","Data":"e594c15c9b46c1317f91fa1f3eb65b2444087bbd86b4ebae9b2708754ccd6aa2"} Nov 01 20:29:06.439047 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:06.439025 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr" event={"ID":"92a72792-1735-4970-88ac-ca2a50a5b63e","Type":"ContainerStarted","Data":"fe104f2f2528f08751fa1f4668202b15803a934d51fe4b18368a9220111a35b7"} Nov 01 20:29:06.459651 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:06.459598 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr" podStartSLOduration=1.459585143 podStartE2EDuration="1.459585143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:29:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:29:06.456705798 +0000 UTC m=+1792.229866311" watchObservedRunningTime="2026-11-01 20:29:06.459585143 +0000 UTC m=+1792.232745654" Nov 01 20:29:06.689668 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:06.689580 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr" Nov 01 20:29:07.690305 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:07.690256 5935 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 01 20:29:07.690706 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:07.690332 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr" podUID="92a72792-1735-4970-88ac-ca2a50a5b63e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 01 20:29:08.689175 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:08.689143 5935 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 01 20:29:08.689359 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:08.689224 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr" podUID="92a72792-1735-4970-88ac-ca2a50a5b63e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 01 20:29:09.693731 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:09.693695 5935 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 01 20:29:09.694370 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:09.694335 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr" podUID="92a72792-1735-4970-88ac-ca2a50a5b63e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 01 20:29:10.688807 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:10.688774 5935 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 01 20:29:10.689002 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:10.688823 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr" podUID="92a72792-1735-4970-88ac-ca2a50a5b63e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 01 20:29:11.689780 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:11.689747 5935 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" start-of-body= Nov 01 20:29:11.690184 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:11.689800 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr" podUID="92a72792-1735-4970-88ac-ca2a50a5b63e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.113:15021/healthz/ready\": dial tcp 10.133.0.113:15021: connect: connection refused" Nov 01 20:29:12.796942 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:12.796914 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr" Nov 01 20:29:12.797789 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:12.797757 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr" Nov 01 20:29:12.799433 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:12.799407 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr" Nov 01 20:29:14.489383 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:14.489341 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr"] Nov 01 20:29:14.490123 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:14.490094 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr" podUID="4659fd62-58a7-4792-ad28-93f3c52c7d3a" containerName="istio-proxy" containerID="cri-o://1a27da7c75ee3916ada810397f40cd896f05191e29efe85b369242c70522e24c" gracePeriod=30 Nov 01 20:29:19.747273 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:19.747247 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr" Nov 01 20:29:19.820643 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:19.820570 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4659fd62-58a7-4792-ad28-93f3c52c7d3a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4659fd62-58a7-4792-ad28-93f3c52c7d3a-workload-socket\") pod \"4659fd62-58a7-4792-ad28-93f3c52c7d3a\" (UID: \"4659fd62-58a7-4792-ad28-93f3c52c7d3a\") " Nov 01 20:29:19.820643 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:19.820630 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4659fd62-58a7-4792-ad28-93f3c52c7d3a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4659fd62-58a7-4792-ad28-93f3c52c7d3a-istio-envoy\") pod \"4659fd62-58a7-4792-ad28-93f3c52c7d3a\" (UID: \"4659fd62-58a7-4792-ad28-93f3c52c7d3a\") " Nov 01 20:29:19.820854 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:19.820684 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4659fd62-58a7-4792-ad28-93f3c52c7d3a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4659fd62-58a7-4792-ad28-93f3c52c7d3a-workload-certs\") pod \"4659fd62-58a7-4792-ad28-93f3c52c7d3a\" (UID: \"4659fd62-58a7-4792-ad28-93f3c52c7d3a\") " Nov 01 20:29:19.820854 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:19.820815 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4659fd62-58a7-4792-ad28-93f3c52c7d3a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4659fd62-58a7-4792-ad28-93f3c52c7d3a-credential-socket\") pod \"4659fd62-58a7-4792-ad28-93f3c52c7d3a\" (UID: \"4659fd62-58a7-4792-ad28-93f3c52c7d3a\") " Nov 01 20:29:19.820854 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:19.820833 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4659fd62-58a7-4792-ad28-93f3c52c7d3a-workload-socket" pod "4659fd62-58a7-4792-ad28-93f3c52c7d3a" (UID: "4659fd62-58a7-4792-ad28-93f3c52c7d3a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:29:19.820854 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:19.820846 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4659fd62-58a7-4792-ad28-93f3c52c7d3a-istio-token\" (UniqueName: \"kubernetes.io/projected/4659fd62-58a7-4792-ad28-93f3c52c7d3a-istio-token\") pod \"4659fd62-58a7-4792-ad28-93f3c52c7d3a\" (UID: \"4659fd62-58a7-4792-ad28-93f3c52c7d3a\") " Nov 01 20:29:19.821060 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:19.820880 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4659fd62-58a7-4792-ad28-93f3c52c7d3a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4659fd62-58a7-4792-ad28-93f3c52c7d3a-istio-podinfo\") pod \"4659fd62-58a7-4792-ad28-93f3c52c7d3a\" (UID: \"4659fd62-58a7-4792-ad28-93f3c52c7d3a\") " Nov 01 20:29:19.821060 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:19.820921 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4659fd62-58a7-4792-ad28-93f3c52c7d3a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4659fd62-58a7-4792-ad28-93f3c52c7d3a-istio-data\") pod \"4659fd62-58a7-4792-ad28-93f3c52c7d3a\" (UID: \"4659fd62-58a7-4792-ad28-93f3c52c7d3a\") " Nov 01 20:29:19.821060 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:19.820949 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4659fd62-58a7-4792-ad28-93f3c52c7d3a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4659fd62-58a7-4792-ad28-93f3c52c7d3a-istiod-ca-cert\") pod \"4659fd62-58a7-4792-ad28-93f3c52c7d3a\" (UID: \"4659fd62-58a7-4792-ad28-93f3c52c7d3a\") " Nov 01 20:29:19.821060 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:19.820985 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4659fd62-58a7-4792-ad28-93f3c52c7d3a-kube-api-access-4pv55\" (UniqueName: \"kubernetes.io/projected/4659fd62-58a7-4792-ad28-93f3c52c7d3a-kube-api-access-4pv55\") pod \"4659fd62-58a7-4792-ad28-93f3c52c7d3a\" (UID: \"4659fd62-58a7-4792-ad28-93f3c52c7d3a\") " Nov 01 20:29:19.821254 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:19.821029 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4659fd62-58a7-4792-ad28-93f3c52c7d3a-workload-certs" pod "4659fd62-58a7-4792-ad28-93f3c52c7d3a" (UID: "4659fd62-58a7-4792-ad28-93f3c52c7d3a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:29:19.821254 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:19.821080 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4659fd62-58a7-4792-ad28-93f3c52c7d3a-credential-socket" pod "4659fd62-58a7-4792-ad28-93f3c52c7d3a" (UID: "4659fd62-58a7-4792-ad28-93f3c52c7d3a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:29:19.821377 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:19.821279 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4659fd62-58a7-4792-ad28-93f3c52c7d3a-istio-data" pod "4659fd62-58a7-4792-ad28-93f3c52c7d3a" (UID: "4659fd62-58a7-4792-ad28-93f3c52c7d3a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:29:19.821443 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:19.821422 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4659fd62-58a7-4792-ad28-93f3c52c7d3a-istiod-ca-cert" pod "4659fd62-58a7-4792-ad28-93f3c52c7d3a" (UID: "4659fd62-58a7-4792-ad28-93f3c52c7d3a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:29:19.821499 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:19.821431 5935 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4659fd62-58a7-4792-ad28-93f3c52c7d3a-credential-socket\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:29:19.821499 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:19.821469 5935 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4659fd62-58a7-4792-ad28-93f3c52c7d3a-istio-data\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:29:19.821499 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:19.821487 5935 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4659fd62-58a7-4792-ad28-93f3c52c7d3a-workload-socket\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:29:19.821715 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:19.821503 5935 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4659fd62-58a7-4792-ad28-93f3c52c7d3a-workload-certs\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:29:19.822881 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:19.822853 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4659fd62-58a7-4792-ad28-93f3c52c7d3a-istio-envoy" pod "4659fd62-58a7-4792-ad28-93f3c52c7d3a" (UID: "4659fd62-58a7-4792-ad28-93f3c52c7d3a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:29:19.823023 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:19.823005 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4659fd62-58a7-4792-ad28-93f3c52c7d3a-istio-podinfo" pod "4659fd62-58a7-4792-ad28-93f3c52c7d3a" (UID: "4659fd62-58a7-4792-ad28-93f3c52c7d3a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:29:19.823134 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:19.823115 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4659fd62-58a7-4792-ad28-93f3c52c7d3a-istio-token" pod "4659fd62-58a7-4792-ad28-93f3c52c7d3a" (UID: "4659fd62-58a7-4792-ad28-93f3c52c7d3a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:29:19.823188 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:19.823134 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4659fd62-58a7-4792-ad28-93f3c52c7d3a-kube-api-access-4pv55" pod "4659fd62-58a7-4792-ad28-93f3c52c7d3a" (UID: "4659fd62-58a7-4792-ad28-93f3c52c7d3a"). InnerVolumeSpecName "kube-api-access-4pv55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:29:19.922767 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:19.922741 5935 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4659fd62-58a7-4792-ad28-93f3c52c7d3a-istio-token\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:29:19.922767 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:19.922765 5935 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4659fd62-58a7-4792-ad28-93f3c52c7d3a-istio-podinfo\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:29:19.922927 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:19.922775 5935 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4659fd62-58a7-4792-ad28-93f3c52c7d3a-istiod-ca-cert\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:29:19.922927 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:19.922784 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pv55\" (UniqueName: \"kubernetes.io/projected/4659fd62-58a7-4792-ad28-93f3c52c7d3a-kube-api-access-4pv55\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:29:19.922927 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:19.922794 5935 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4659fd62-58a7-4792-ad28-93f3c52c7d3a-istio-envoy\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:29:20.556961 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:20.556925 5935 generic.go:356] "Generic (PLEG): container finished" podID="4659fd62-58a7-4792-ad28-93f3c52c7d3a" containerID="1a27da7c75ee3916ada810397f40cd896f05191e29efe85b369242c70522e24c" exitCode=0 Nov 01 20:29:20.557148 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:20.556974 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr" Nov 01 20:29:20.557148 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:20.557014 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr" event={"ID":"4659fd62-58a7-4792-ad28-93f3c52c7d3a","Type":"ContainerDied","Data":"1a27da7c75ee3916ada810397f40cd896f05191e29efe85b369242c70522e24c"} Nov 01 20:29:20.557148 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:20.557080 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr" event={"ID":"4659fd62-58a7-4792-ad28-93f3c52c7d3a","Type":"ContainerDied","Data":"4b92b864859bce30daf499d2bf456248ad5a67801c762de07127833dfac5cdea"} Nov 01 20:29:20.557148 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:20.557110 5935 scope.go:122] "RemoveContainer" containerID="1a27da7c75ee3916ada810397f40cd896f05191e29efe85b369242c70522e24c" Nov 01 20:29:20.567815 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:20.567793 5935 scope.go:122] "RemoveContainer" containerID="1a27da7c75ee3916ada810397f40cd896f05191e29efe85b369242c70522e24c" Nov 01 20:29:20.568091 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:29:20.568073 5935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a27da7c75ee3916ada810397f40cd896f05191e29efe85b369242c70522e24c\": container with ID starting with 1a27da7c75ee3916ada810397f40cd896f05191e29efe85b369242c70522e24c not found: ID does not exist" containerID="1a27da7c75ee3916ada810397f40cd896f05191e29efe85b369242c70522e24c" Nov 01 20:29:20.568154 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:20.568102 5935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a27da7c75ee3916ada810397f40cd896f05191e29efe85b369242c70522e24c"} err="failed to get container status \"1a27da7c75ee3916ada810397f40cd896f05191e29efe85b369242c70522e24c\": rpc error: code = NotFound desc = could not find container \"1a27da7c75ee3916ada810397f40cd896f05191e29efe85b369242c70522e24c\": container with ID starting with 1a27da7c75ee3916ada810397f40cd896f05191e29efe85b369242c70522e24c not found: ID does not exist" Nov 01 20:29:20.576123 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:20.576080 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr"] Nov 01 20:29:20.579392 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:20.579370 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-9de2ac-openshift-default-5cccf5d68c-6smnr"] Nov 01 20:29:22.352978 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:22.352938 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4659fd62-58a7-4792-ad28-93f3c52c7d3a" path="/var/lib/kubelet/pods/4659fd62-58a7-4792-ad28-93f3c52c7d3a/volumes" Nov 01 20:29:35.843771 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:29:35.843735 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:29:55.250077 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:55.250039 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd"] Nov 01 20:29:55.250880 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:55.250857 5935 cpu_manager.go:414] "containerMap: removing container" podUID="4659fd62-58a7-4792-ad28-93f3c52c7d3a" containerName="istio-proxy" Nov 01 20:29:55.250975 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:55.250883 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4659fd62-58a7-4792-ad28-93f3c52c7d3a" containerName="istio-proxy" Nov 01 20:29:55.251049 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:55.251035 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="4659fd62-58a7-4792-ad28-93f3c52c7d3a" containerName="istio-proxy" Nov 01 20:29:55.254574 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:55.254550 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd" Nov 01 20:29:55.260139 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:55.259585 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-036c69-openshift-default-dockercfg-9njcd\"" Nov 01 20:29:55.261590 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:55.261568 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd"] Nov 01 20:29:55.299794 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:55.299760 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cb9e245-7141-402d-9601-6f374e26fff3-istio-data\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd\" (UID: \"5cb9e245-7141-402d-9601-6f374e26fff3\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd" Nov 01 20:29:55.299992 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:55.299887 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cb9e245-7141-402d-9601-6f374e26fff3-istio-envoy\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd\" (UID: \"5cb9e245-7141-402d-9601-6f374e26fff3\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd" Nov 01 20:29:55.299992 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:55.299915 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cb9e245-7141-402d-9601-6f374e26fff3-credential-socket\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd\" (UID: \"5cb9e245-7141-402d-9601-6f374e26fff3\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd" Nov 01 20:29:55.299992 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:55.299975 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cb9e245-7141-402d-9601-6f374e26fff3-istio-podinfo\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd\" (UID: \"5cb9e245-7141-402d-9601-6f374e26fff3\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd" Nov 01 20:29:55.300173 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:55.300016 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k6t6\" (UniqueName: \"kubernetes.io/projected/5cb9e245-7141-402d-9601-6f374e26fff3-kube-api-access-9k6t6\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd\" (UID: \"5cb9e245-7141-402d-9601-6f374e26fff3\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd" Nov 01 20:29:55.300173 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:55.300126 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cb9e245-7141-402d-9601-6f374e26fff3-istio-token\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd\" (UID: \"5cb9e245-7141-402d-9601-6f374e26fff3\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd" Nov 01 20:29:55.300173 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:55.300156 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cb9e245-7141-402d-9601-6f374e26fff3-workload-certs\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd\" (UID: \"5cb9e245-7141-402d-9601-6f374e26fff3\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd" Nov 01 20:29:55.300350 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:55.300194 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cb9e245-7141-402d-9601-6f374e26fff3-workload-socket\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd\" (UID: \"5cb9e245-7141-402d-9601-6f374e26fff3\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd" Nov 01 20:29:55.300350 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:55.300318 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cb9e245-7141-402d-9601-6f374e26fff3-istiod-ca-cert\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd\" (UID: \"5cb9e245-7141-402d-9601-6f374e26fff3\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd" Nov 01 20:29:55.400897 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:55.400870 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cb9e245-7141-402d-9601-6f374e26fff3-istio-token\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd\" (UID: \"5cb9e245-7141-402d-9601-6f374e26fff3\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd" Nov 01 20:29:55.401059 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:55.400907 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cb9e245-7141-402d-9601-6f374e26fff3-workload-certs\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd\" (UID: \"5cb9e245-7141-402d-9601-6f374e26fff3\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd" Nov 01 20:29:55.401059 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:55.401042 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cb9e245-7141-402d-9601-6f374e26fff3-workload-socket\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd\" (UID: \"5cb9e245-7141-402d-9601-6f374e26fff3\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd" Nov 01 20:29:55.401183 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:55.401115 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cb9e245-7141-402d-9601-6f374e26fff3-istiod-ca-cert\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd\" (UID: \"5cb9e245-7141-402d-9601-6f374e26fff3\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd" Nov 01 20:29:55.401280 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:55.401246 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cb9e245-7141-402d-9601-6f374e26fff3-istio-data\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd\" (UID: \"5cb9e245-7141-402d-9601-6f374e26fff3\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd" Nov 01 20:29:55.401394 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:55.401332 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cb9e245-7141-402d-9601-6f374e26fff3-workload-certs\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd\" (UID: \"5cb9e245-7141-402d-9601-6f374e26fff3\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd" Nov 01 20:29:55.401394 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:55.401363 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cb9e245-7141-402d-9601-6f374e26fff3-istio-envoy\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd\" (UID: \"5cb9e245-7141-402d-9601-6f374e26fff3\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd" Nov 01 20:29:55.401503 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:55.401396 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cb9e245-7141-402d-9601-6f374e26fff3-credential-socket\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd\" (UID: \"5cb9e245-7141-402d-9601-6f374e26fff3\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd" Nov 01 20:29:55.401503 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:55.401423 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cb9e245-7141-402d-9601-6f374e26fff3-workload-socket\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd\" (UID: \"5cb9e245-7141-402d-9601-6f374e26fff3\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd" Nov 01 20:29:55.401503 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:55.401427 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cb9e245-7141-402d-9601-6f374e26fff3-istio-podinfo\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd\" (UID: \"5cb9e245-7141-402d-9601-6f374e26fff3\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd" Nov 01 20:29:55.401659 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:55.401502 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k6t6\" (UniqueName: \"kubernetes.io/projected/5cb9e245-7141-402d-9601-6f374e26fff3-kube-api-access-9k6t6\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd\" (UID: \"5cb9e245-7141-402d-9601-6f374e26fff3\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd" Nov 01 20:29:55.401659 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:55.401603 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cb9e245-7141-402d-9601-6f374e26fff3-istio-data\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd\" (UID: \"5cb9e245-7141-402d-9601-6f374e26fff3\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd" Nov 01 20:29:55.401902 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:55.401881 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cb9e245-7141-402d-9601-6f374e26fff3-istiod-ca-cert\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd\" (UID: \"5cb9e245-7141-402d-9601-6f374e26fff3\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd" Nov 01 20:29:55.402062 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:55.402042 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cb9e245-7141-402d-9601-6f374e26fff3-credential-socket\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd\" (UID: \"5cb9e245-7141-402d-9601-6f374e26fff3\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd" Nov 01 20:29:55.403621 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:55.403600 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cb9e245-7141-402d-9601-6f374e26fff3-istio-envoy\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd\" (UID: \"5cb9e245-7141-402d-9601-6f374e26fff3\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd" Nov 01 20:29:55.403912 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:55.403895 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cb9e245-7141-402d-9601-6f374e26fff3-istio-podinfo\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd\" (UID: \"5cb9e245-7141-402d-9601-6f374e26fff3\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd" Nov 01 20:29:55.409024 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:55.408992 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cb9e245-7141-402d-9601-6f374e26fff3-istio-token\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd\" (UID: \"5cb9e245-7141-402d-9601-6f374e26fff3\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd" Nov 01 20:29:55.409456 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:55.409438 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k6t6\" (UniqueName: \"kubernetes.io/projected/5cb9e245-7141-402d-9601-6f374e26fff3-kube-api-access-9k6t6\") pod \"e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd\" (UID: \"5cb9e245-7141-402d-9601-6f374e26fff3\") " pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd" Nov 01 20:29:55.571563 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:55.571487 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd" Nov 01 20:29:55.850513 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:55.850486 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd"] Nov 01 20:29:55.852330 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:29:55.852281 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cb9e245_7141_402d_9601_6f374e26fff3.slice/crio-3a4c243b3019e051b910e6c94b6d910ffb138a9eee38d33ea13753e8328ffe23 WatchSource:0}: Error finding container 3a4c243b3019e051b910e6c94b6d910ffb138a9eee38d33ea13753e8328ffe23: Status 404 returned error can't find the container with id 3a4c243b3019e051b910e6c94b6d910ffb138a9eee38d33ea13753e8328ffe23 Nov 01 20:29:55.854848 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:55.854808 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:29:55.854932 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:55.854895 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:29:55.854972 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:55.854942 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:29:56.785932 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:56.785890 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd" event={"ID":"5cb9e245-7141-402d-9601-6f374e26fff3","Type":"ContainerStarted","Data":"56a4682aedcc5a3c462dd7638b191dea7f2bb66633345428601b436421a05e13"} Nov 01 20:29:56.786370 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:56.785954 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd" event={"ID":"5cb9e245-7141-402d-9601-6f374e26fff3","Type":"ContainerStarted","Data":"3a4c243b3019e051b910e6c94b6d910ffb138a9eee38d33ea13753e8328ffe23"} Nov 01 20:29:56.804126 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:56.804078 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd" podStartSLOduration=1.804065415 podStartE2EDuration="1.804065415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:29:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:29:56.802545941 +0000 UTC m=+1842.575706468" watchObservedRunningTime="2026-11-01 20:29:56.804065415 +0000 UTC m=+1842.577226003" Nov 01 20:29:57.572136 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:57.572104 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd" Nov 01 20:29:57.573972 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:57.573947 5935 patch_prober.go:29] interesting pod/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Nov 01 20:29:57.574107 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:57.574000 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd" podUID="5cb9e245-7141-402d-9601-6f374e26fff3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Nov 01 20:29:58.572717 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:58.572686 5935 patch_prober.go:29] interesting pod/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Nov 01 20:29:58.573089 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:58.572735 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd" podUID="5cb9e245-7141-402d-9601-6f374e26fff3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Nov 01 20:29:59.572771 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:59.572739 5935 patch_prober.go:29] interesting pod/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Nov 01 20:29:59.573177 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:29:59.572789 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd" podUID="5cb9e245-7141-402d-9601-6f374e26fff3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Nov 01 20:30:00.189528 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:00.189489 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d2dc0ad5-29892750-rm8nd"] Nov 01 20:30:00.194679 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:00.194656 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d2dc0ad5-29892750-rm8nd" Nov 01 20:30:00.202192 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:00.202161 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d2dc0ad5-29892750-rm8nd"] Nov 01 20:30:00.273384 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:00.260605 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\"" Nov 01 20:30:00.285237 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:00.285172 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5b342ce-508a-445c-8034-ff83d26a7190-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-d2dc0ad5-29892750-rm8nd\" (UID: \"c5b342ce-508a-445c-8034-ff83d26a7190\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d2dc0ad5-29892750-rm8nd" Nov 01 20:30:00.285237 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:00.285231 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb87r\" (UniqueName: \"kubernetes.io/projected/c5b342ce-508a-445c-8034-ff83d26a7190-kube-api-access-xb87r\") pod \"maas-api-key-cleanup-e2e-mt-d2dc0ad5-29892750-rm8nd\" (UID: \"c5b342ce-508a-445c-8034-ff83d26a7190\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d2dc0ad5-29892750-rm8nd" Nov 01 20:30:00.386656 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:00.386626 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5b342ce-508a-445c-8034-ff83d26a7190-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-d2dc0ad5-29892750-rm8nd\" (UID: \"c5b342ce-508a-445c-8034-ff83d26a7190\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d2dc0ad5-29892750-rm8nd" Nov 01 20:30:00.386861 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:00.386764 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb87r\" (UniqueName: \"kubernetes.io/projected/c5b342ce-508a-445c-8034-ff83d26a7190-kube-api-access-xb87r\") pod \"maas-api-key-cleanup-e2e-mt-d2dc0ad5-29892750-rm8nd\" (UID: \"c5b342ce-508a-445c-8034-ff83d26a7190\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d2dc0ad5-29892750-rm8nd" Nov 01 20:30:00.387365 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:00.387332 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5b342ce-508a-445c-8034-ff83d26a7190-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-d2dc0ad5-29892750-rm8nd\" (UID: \"c5b342ce-508a-445c-8034-ff83d26a7190\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d2dc0ad5-29892750-rm8nd" Nov 01 20:30:00.394485 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:00.394466 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb87r\" (UniqueName: \"kubernetes.io/projected/c5b342ce-508a-445c-8034-ff83d26a7190-kube-api-access-xb87r\") pod \"maas-api-key-cleanup-e2e-mt-d2dc0ad5-29892750-rm8nd\" (UID: \"c5b342ce-508a-445c-8034-ff83d26a7190\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d2dc0ad5-29892750-rm8nd" Nov 01 20:30:00.513713 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:00.513685 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d2dc0ad5-29892750-rm8nd" Nov 01 20:30:00.572116 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:00.572079 5935 patch_prober.go:29] interesting pod/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Nov 01 20:30:00.572321 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:00.572152 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd" podUID="5cb9e245-7141-402d-9601-6f374e26fff3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Nov 01 20:30:00.819511 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:00.819483 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d2dc0ad5-29892750-rm8nd"] Nov 01 20:30:00.821321 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:30:00.821273 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5b342ce_508a_445c_8034_ff83d26a7190.slice/crio-a2645bf82a1448416c3ca1fcf8d8e0c37b9e0cbd2291fe3045c291f8be40b8b5 WatchSource:0}: Error finding container a2645bf82a1448416c3ca1fcf8d8e0c37b9e0cbd2291fe3045c291f8be40b8b5: Status 404 returned error can't find the container with id a2645bf82a1448416c3ca1fcf8d8e0c37b9e0cbd2291fe3045c291f8be40b8b5 Nov 01 20:30:01.572258 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:01.572226 5935 patch_prober.go:29] interesting pod/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" start-of-body= Nov 01 20:30:01.572457 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:01.572278 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd" podUID="5cb9e245-7141-402d-9601-6f374e26fff3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.114:15021/healthz/ready\": dial tcp 10.133.0.114:15021: connect: connection refused" Nov 01 20:30:01.821043 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:01.821004 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d2dc0ad5-29892750-rm8nd" event={"ID":"c5b342ce-508a-445c-8034-ff83d26a7190","Type":"ContainerStarted","Data":"0f576144ecf0fbb4b00f04a7954a9939b40af2969fe2948f11e27d08aec977f5"} Nov 01 20:30:01.821545 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:01.821071 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d2dc0ad5-29892750-rm8nd" event={"ID":"c5b342ce-508a-445c-8034-ff83d26a7190","Type":"ContainerStarted","Data":"a2645bf82a1448416c3ca1fcf8d8e0c37b9e0cbd2291fe3045c291f8be40b8b5"} Nov 01 20:30:01.838809 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:01.838655 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d2dc0ad5-29892750-rm8nd" podStartSLOduration=1.8386323 podStartE2EDuration="1.8386323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:30:01.8376815 +0000 UTC m=+1847.610842025" watchObservedRunningTime="2026-11-01 20:30:01.8386323 +0000 UTC m=+1847.611792856" Nov 01 20:30:02.576436 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:02.576406 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd" Nov 01 20:30:02.576736 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:02.576715 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd" Nov 01 20:30:02.577435 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:02.577419 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd" Nov 01 20:30:02.829755 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:02.829650 5935 generic.go:356] "Generic (PLEG): container finished" podID="c5b342ce-508a-445c-8034-ff83d26a7190" containerID="0f576144ecf0fbb4b00f04a7954a9939b40af2969fe2948f11e27d08aec977f5" exitCode=60 Nov 01 20:30:02.829755 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:02.829733 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d2dc0ad5-29892750-rm8nd" event={"ID":"c5b342ce-508a-445c-8034-ff83d26a7190","Type":"ContainerDied","Data":"0f576144ecf0fbb4b00f04a7954a9939b40af2969fe2948f11e27d08aec977f5"} Nov 01 20:30:02.830383 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:02.830084 5935 scope.go:122] "RemoveContainer" containerID="0f576144ecf0fbb4b00f04a7954a9939b40af2969fe2948f11e27d08aec977f5" Nov 01 20:30:03.844168 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:03.839705 5935 generic.go:356] "Generic (PLEG): container finished" podID="c5b342ce-508a-445c-8034-ff83d26a7190" containerID="f80470b746c8e11ca2130ea1810d3acb48e43ffa34ec98dd6e76dfe1b1019687" exitCode=60 Nov 01 20:30:03.844168 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:03.840912 5935 scope.go:122] "RemoveContainer" containerID="f80470b746c8e11ca2130ea1810d3acb48e43ffa34ec98dd6e76dfe1b1019687" Nov 01 20:30:03.844168 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:30:03.841189 5935 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-d2dc0ad5-29892750-rm8nd_odh-ai-gateway-infra(c5b342ce-508a-445c-8034-ff83d26a7190)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d2dc0ad5-29892750-rm8nd" podUID="c5b342ce-508a-445c-8034-ff83d26a7190" Nov 01 20:30:03.844168 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:03.841477 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d2dc0ad5-29892750-rm8nd" event={"ID":"c5b342ce-508a-445c-8034-ff83d26a7190","Type":"ContainerDied","Data":"f80470b746c8e11ca2130ea1810d3acb48e43ffa34ec98dd6e76dfe1b1019687"} Nov 01 20:30:03.844168 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:03.841564 5935 scope.go:122] "RemoveContainer" containerID="0f576144ecf0fbb4b00f04a7954a9939b40af2969fe2948f11e27d08aec977f5" Nov 01 20:30:04.849069 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:04.849040 5935 scope.go:122] "RemoveContainer" containerID="f80470b746c8e11ca2130ea1810d3acb48e43ffa34ec98dd6e76dfe1b1019687" Nov 01 20:30:04.849506 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:30:04.849310 5935 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-d2dc0ad5-29892750-rm8nd_odh-ai-gateway-infra(c5b342ce-508a-445c-8034-ff83d26a7190)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d2dc0ad5-29892750-rm8nd" podUID="c5b342ce-508a-445c-8034-ff83d26a7190" Nov 01 20:30:05.845660 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:30:05.845620 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:30:18.355272 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:18.355239 5935 scope.go:122] "RemoveContainer" containerID="f80470b746c8e11ca2130ea1810d3acb48e43ffa34ec98dd6e76dfe1b1019687" Nov 01 20:30:18.940189 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:18.940063 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d2dc0ad5-29892750-rm8nd" event={"ID":"c5b342ce-508a-445c-8034-ff83d26a7190","Type":"ContainerStarted","Data":"efa717fd8b309de1150b7eaefb0428c9247bf16e8a31d72960c1fedc303b6213"} Nov 01 20:30:19.947768 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:19.947730 5935 generic.go:356] "Generic (PLEG): container finished" podID="c5b342ce-508a-445c-8034-ff83d26a7190" containerID="efa717fd8b309de1150b7eaefb0428c9247bf16e8a31d72960c1fedc303b6213" exitCode=60 Nov 01 20:30:19.948274 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:19.947796 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d2dc0ad5-29892750-rm8nd" event={"ID":"c5b342ce-508a-445c-8034-ff83d26a7190","Type":"ContainerDied","Data":"efa717fd8b309de1150b7eaefb0428c9247bf16e8a31d72960c1fedc303b6213"} Nov 01 20:30:19.948274 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:19.947855 5935 scope.go:122] "RemoveContainer" containerID="f80470b746c8e11ca2130ea1810d3acb48e43ffa34ec98dd6e76dfe1b1019687" Nov 01 20:30:19.948443 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:19.948275 5935 scope.go:122] "RemoveContainer" containerID="efa717fd8b309de1150b7eaefb0428c9247bf16e8a31d72960c1fedc303b6213" Nov 01 20:30:19.948666 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:30:19.948643 5935 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-d2dc0ad5-29892750-rm8nd_odh-ai-gateway-infra(c5b342ce-508a-445c-8034-ff83d26a7190)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d2dc0ad5-29892750-rm8nd" podUID="c5b342ce-508a-445c-8034-ff83d26a7190" Nov 01 20:30:19.965381 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:19.965352 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d2dc0ad5-29892750-rm8nd"] Nov 01 20:30:21.102978 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:21.102953 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d2dc0ad5-29892750-rm8nd" Nov 01 20:30:21.250212 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:21.250111 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5b342ce-508a-445c-8034-ff83d26a7190-maas-api-service-ca" pod "c5b342ce-508a-445c-8034-ff83d26a7190" (UID: "c5b342ce-508a-445c-8034-ff83d26a7190"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:30:21.250469 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:21.250445 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5b342ce-508a-445c-8034-ff83d26a7190-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5b342ce-508a-445c-8034-ff83d26a7190-maas-api-service-ca\") pod \"c5b342ce-508a-445c-8034-ff83d26a7190\" (UID: \"c5b342ce-508a-445c-8034-ff83d26a7190\") " Nov 01 20:30:21.250716 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:21.250699 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5b342ce-508a-445c-8034-ff83d26a7190-kube-api-access-xb87r\" (UniqueName: \"kubernetes.io/projected/c5b342ce-508a-445c-8034-ff83d26a7190-kube-api-access-xb87r\") pod \"c5b342ce-508a-445c-8034-ff83d26a7190\" (UID: \"c5b342ce-508a-445c-8034-ff83d26a7190\") " Nov 01 20:30:21.253184 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:21.253159 5935 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5b342ce-508a-445c-8034-ff83d26a7190-maas-api-service-ca\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:30:21.266989 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:21.266934 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5b342ce-508a-445c-8034-ff83d26a7190-kube-api-access-xb87r" pod "c5b342ce-508a-445c-8034-ff83d26a7190" (UID: "c5b342ce-508a-445c-8034-ff83d26a7190"). InnerVolumeSpecName "kube-api-access-xb87r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:30:21.354438 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:21.354400 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xb87r\" (UniqueName: \"kubernetes.io/projected/c5b342ce-508a-445c-8034-ff83d26a7190-kube-api-access-xb87r\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:30:21.965205 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:21.965169 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d2dc0ad5-29892750-rm8nd" event={"ID":"c5b342ce-508a-445c-8034-ff83d26a7190","Type":"ContainerDied","Data":"a2645bf82a1448416c3ca1fcf8d8e0c37b9e0cbd2291fe3045c291f8be40b8b5"} Nov 01 20:30:21.965443 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:21.965212 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d2dc0ad5-29892750-rm8nd" Nov 01 20:30:21.965443 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:21.965239 5935 scope.go:122] "RemoveContainer" containerID="efa717fd8b309de1150b7eaefb0428c9247bf16e8a31d72960c1fedc303b6213" Nov 01 20:30:21.995886 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:21.995856 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d2dc0ad5-29892750-rm8nd"] Nov 01 20:30:22.001296 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:22.001256 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-d2dc0ad5-29892750-rm8nd"] Nov 01 20:30:22.353939 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:22.353855 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5b342ce-508a-445c-8034-ff83d26a7190" path="/var/lib/kubelet/pods/c5b342ce-508a-445c-8034-ff83d26a7190/volumes" Nov 01 20:30:34.361577 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:34.361542 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-22g4n"] Nov 01 20:30:34.362244 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:34.361745 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-22g4n" podUID="52680409-4495-4000-ade4-4498da85c23b" containerName="discovery" containerID="cri-o://022a2f151e43c1345eca22d401d2e1a961833cf4e03cb450b926cbaa5b8afffb" gracePeriod=30 Nov 01 20:30:34.644142 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:34.644117 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-22g4n" Nov 01 20:30:34.712777 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:34.712738 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52680409-4495-4000-ade4-4498da85c23b-istio-token\" (UniqueName: \"kubernetes.io/projected/52680409-4495-4000-ade4-4498da85c23b-istio-token\") pod \"52680409-4495-4000-ade4-4498da85c23b\" (UID: \"52680409-4495-4000-ade4-4498da85c23b\") " Nov 01 20:30:34.712970 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:34.712796 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52680409-4495-4000-ade4-4498da85c23b-cacerts\" (UniqueName: \"kubernetes.io/secret/52680409-4495-4000-ade4-4498da85c23b-cacerts\") pod \"52680409-4495-4000-ade4-4498da85c23b\" (UID: \"52680409-4495-4000-ade4-4498da85c23b\") " Nov 01 20:30:34.712970 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:34.712856 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52680409-4495-4000-ade4-4498da85c23b-kube-api-access-zhh6r\" (UniqueName: \"kubernetes.io/projected/52680409-4495-4000-ade4-4498da85c23b-kube-api-access-zhh6r\") pod \"52680409-4495-4000-ade4-4498da85c23b\" (UID: \"52680409-4495-4000-ade4-4498da85c23b\") " Nov 01 20:30:34.712970 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:34.712902 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52680409-4495-4000-ade4-4498da85c23b-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/52680409-4495-4000-ade4-4498da85c23b-istio-csr-dns-cert\") pod \"52680409-4495-4000-ade4-4498da85c23b\" (UID: \"52680409-4495-4000-ade4-4498da85c23b\") " Nov 01 20:30:34.712970 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:34.712942 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52680409-4495-4000-ade4-4498da85c23b-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/52680409-4495-4000-ade4-4498da85c23b-istio-csr-ca-configmap\") pod \"52680409-4495-4000-ade4-4498da85c23b\" (UID: \"52680409-4495-4000-ade4-4498da85c23b\") " Nov 01 20:30:34.713273 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:34.712998 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52680409-4495-4000-ade4-4498da85c23b-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/52680409-4495-4000-ade4-4498da85c23b-istio-kubeconfig\") pod \"52680409-4495-4000-ade4-4498da85c23b\" (UID: \"52680409-4495-4000-ade4-4498da85c23b\") " Nov 01 20:30:34.713273 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:34.713116 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52680409-4495-4000-ade4-4498da85c23b-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/52680409-4495-4000-ade4-4498da85c23b-local-certs\") pod \"52680409-4495-4000-ade4-4498da85c23b\" (UID: \"52680409-4495-4000-ade4-4498da85c23b\") " Nov 01 20:30:34.713890 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:34.713839 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52680409-4495-4000-ade4-4498da85c23b-istio-csr-ca-configmap" pod "52680409-4495-4000-ade4-4498da85c23b" (UID: "52680409-4495-4000-ade4-4498da85c23b"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:30:34.716105 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:34.716071 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52680409-4495-4000-ade4-4498da85c23b-istio-kubeconfig" pod "52680409-4495-4000-ade4-4498da85c23b" (UID: "52680409-4495-4000-ade4-4498da85c23b"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:30:34.716234 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:34.716103 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52680409-4495-4000-ade4-4498da85c23b-local-certs" pod "52680409-4495-4000-ade4-4498da85c23b" (UID: "52680409-4495-4000-ade4-4498da85c23b"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:30:34.716635 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:34.716611 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52680409-4495-4000-ade4-4498da85c23b-cacerts" pod "52680409-4495-4000-ade4-4498da85c23b" (UID: "52680409-4495-4000-ade4-4498da85c23b"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:30:34.716832 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:34.716795 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52680409-4495-4000-ade4-4498da85c23b-istio-csr-dns-cert" pod "52680409-4495-4000-ade4-4498da85c23b" (UID: "52680409-4495-4000-ade4-4498da85c23b"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:30:34.716931 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:34.716907 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52680409-4495-4000-ade4-4498da85c23b-kube-api-access-zhh6r" pod "52680409-4495-4000-ade4-4498da85c23b" (UID: "52680409-4495-4000-ade4-4498da85c23b"). InnerVolumeSpecName "kube-api-access-zhh6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:30:34.717738 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:34.717712 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52680409-4495-4000-ade4-4498da85c23b-istio-token" pod "52680409-4495-4000-ade4-4498da85c23b" (UID: "52680409-4495-4000-ade4-4498da85c23b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:30:34.814440 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:34.814401 5935 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52680409-4495-4000-ade4-4498da85c23b-istio-token\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:30:34.814440 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:34.814430 5935 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/52680409-4495-4000-ade4-4498da85c23b-cacerts\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:30:34.814440 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:34.814441 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhh6r\" (UniqueName: \"kubernetes.io/projected/52680409-4495-4000-ade4-4498da85c23b-kube-api-access-zhh6r\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:30:34.814440 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:34.814451 5935 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/52680409-4495-4000-ade4-4498da85c23b-istio-csr-dns-cert\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:30:34.814725 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:34.814460 5935 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/52680409-4495-4000-ade4-4498da85c23b-istio-csr-ca-configmap\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:30:34.814725 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:34.814469 5935 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/52680409-4495-4000-ade4-4498da85c23b-istio-kubeconfig\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:30:34.814725 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:34.814478 5935 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/52680409-4495-4000-ade4-4498da85c23b-local-certs\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:30:35.051077 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:35.051042 5935 generic.go:356] "Generic (PLEG): container finished" podID="52680409-4495-4000-ade4-4498da85c23b" containerID="022a2f151e43c1345eca22d401d2e1a961833cf4e03cb450b926cbaa5b8afffb" exitCode=0 Nov 01 20:30:35.051306 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:35.051093 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-22g4n" Nov 01 20:30:35.051306 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:35.051127 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-22g4n" event={"ID":"52680409-4495-4000-ade4-4498da85c23b","Type":"ContainerDied","Data":"022a2f151e43c1345eca22d401d2e1a961833cf4e03cb450b926cbaa5b8afffb"} Nov 01 20:30:35.051306 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:35.051182 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-22g4n" event={"ID":"52680409-4495-4000-ade4-4498da85c23b","Type":"ContainerDied","Data":"b96980928473bf297ebf88c95152acc0beaeb13b491bd6fd16e73bd6b45ce35e"} Nov 01 20:30:35.051306 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:35.051209 5935 scope.go:122] "RemoveContainer" containerID="022a2f151e43c1345eca22d401d2e1a961833cf4e03cb450b926cbaa5b8afffb" Nov 01 20:30:35.062661 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:35.062643 5935 scope.go:122] "RemoveContainer" containerID="022a2f151e43c1345eca22d401d2e1a961833cf4e03cb450b926cbaa5b8afffb" Nov 01 20:30:35.062981 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:30:35.062962 5935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"022a2f151e43c1345eca22d401d2e1a961833cf4e03cb450b926cbaa5b8afffb\": container with ID starting with 022a2f151e43c1345eca22d401d2e1a961833cf4e03cb450b926cbaa5b8afffb not found: ID does not exist" containerID="022a2f151e43c1345eca22d401d2e1a961833cf4e03cb450b926cbaa5b8afffb" Nov 01 20:30:35.063079 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:35.062993 5935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"022a2f151e43c1345eca22d401d2e1a961833cf4e03cb450b926cbaa5b8afffb"} err="failed to get container status \"022a2f151e43c1345eca22d401d2e1a961833cf4e03cb450b926cbaa5b8afffb\": rpc error: code = NotFound desc = could not find container \"022a2f151e43c1345eca22d401d2e1a961833cf4e03cb450b926cbaa5b8afffb\": container with ID starting with 022a2f151e43c1345eca22d401d2e1a961833cf4e03cb450b926cbaa5b8afffb not found: ID does not exist" Nov 01 20:30:35.076370 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:35.076339 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-22g4n"] Nov 01 20:30:35.080108 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:35.080079 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-22g4n"] Nov 01 20:30:35.849485 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:30:35.849415 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:30:36.353428 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:36.353394 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52680409-4495-4000-ade4-4498da85c23b" path="/var/lib/kubelet/pods/52680409-4495-4000-ade4-4498da85c23b/volumes" Nov 01 20:30:37.888728 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:37.888683 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d2dc0ad5-ff87549c8-xs57b"] Nov 01 20:30:37.889250 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:37.888957 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d2dc0ad5-ff87549c8-xs57b" podUID="7cdc5a94-7c68-49f2-95b9-8d23d1355581" containerName="maas-api" containerID="cri-o://ca9fede5f97956aa16b8ff2e5d5f554da676983120099d42db6d91dc25846b5f" gracePeriod=30 Nov 01 20:30:38.076504 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:38.076467 5935 generic.go:356] "Generic (PLEG): container finished" podID="7cdc5a94-7c68-49f2-95b9-8d23d1355581" containerID="ca9fede5f97956aa16b8ff2e5d5f554da676983120099d42db6d91dc25846b5f" exitCode=0 Nov 01 20:30:38.076984 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:38.076745 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d2dc0ad5-ff87549c8-xs57b" event={"ID":"7cdc5a94-7c68-49f2-95b9-8d23d1355581","Type":"ContainerDied","Data":"ca9fede5f97956aa16b8ff2e5d5f554da676983120099d42db6d91dc25846b5f"} Nov 01 20:30:38.216372 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:38.216339 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d2dc0ad5-ff87549c8-xs57b" Nov 01 20:30:38.262330 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:38.262250 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cdc5a94-7c68-49f2-95b9-8d23d1355581-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cdc5a94-7c68-49f2-95b9-8d23d1355581-maas-api-tls\") pod \"7cdc5a94-7c68-49f2-95b9-8d23d1355581\" (UID: \"7cdc5a94-7c68-49f2-95b9-8d23d1355581\") " Nov 01 20:30:38.262330 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:38.262325 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cdc5a94-7c68-49f2-95b9-8d23d1355581-kube-api-access-czkq2\" (UniqueName: \"kubernetes.io/projected/7cdc5a94-7c68-49f2-95b9-8d23d1355581-kube-api-access-czkq2\") pod \"7cdc5a94-7c68-49f2-95b9-8d23d1355581\" (UID: \"7cdc5a94-7c68-49f2-95b9-8d23d1355581\") " Nov 01 20:30:38.262602 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:38.262427 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cdc5a94-7c68-49f2-95b9-8d23d1355581-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cdc5a94-7c68-49f2-95b9-8d23d1355581-openshift-service-ca\") pod \"7cdc5a94-7c68-49f2-95b9-8d23d1355581\" (UID: \"7cdc5a94-7c68-49f2-95b9-8d23d1355581\") " Nov 01 20:30:38.262602 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:38.262502 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cdc5a94-7c68-49f2-95b9-8d23d1355581-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cdc5a94-7c68-49f2-95b9-8d23d1355581-metrics-certs\") pod \"7cdc5a94-7c68-49f2-95b9-8d23d1355581\" (UID: \"7cdc5a94-7c68-49f2-95b9-8d23d1355581\") " Nov 01 20:30:38.262875 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:38.262806 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cdc5a94-7c68-49f2-95b9-8d23d1355581-openshift-service-ca" pod "7cdc5a94-7c68-49f2-95b9-8d23d1355581" (UID: "7cdc5a94-7c68-49f2-95b9-8d23d1355581"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:30:38.263180 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:38.263154 5935 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7cdc5a94-7c68-49f2-95b9-8d23d1355581-openshift-service-ca\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:30:38.264641 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:38.264616 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cdc5a94-7c68-49f2-95b9-8d23d1355581-metrics-certs" pod "7cdc5a94-7c68-49f2-95b9-8d23d1355581" (UID: "7cdc5a94-7c68-49f2-95b9-8d23d1355581"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:30:38.264771 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:38.264754 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cdc5a94-7c68-49f2-95b9-8d23d1355581-maas-api-tls" pod "7cdc5a94-7c68-49f2-95b9-8d23d1355581" (UID: "7cdc5a94-7c68-49f2-95b9-8d23d1355581"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:30:38.264935 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:38.264915 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cdc5a94-7c68-49f2-95b9-8d23d1355581-kube-api-access-czkq2" pod "7cdc5a94-7c68-49f2-95b9-8d23d1355581" (UID: "7cdc5a94-7c68-49f2-95b9-8d23d1355581"). InnerVolumeSpecName "kube-api-access-czkq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:30:38.363968 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:38.363932 5935 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7cdc5a94-7c68-49f2-95b9-8d23d1355581-maas-api-tls\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:30:38.363968 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:38.363969 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czkq2\" (UniqueName: \"kubernetes.io/projected/7cdc5a94-7c68-49f2-95b9-8d23d1355581-kube-api-access-czkq2\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:30:38.364175 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:38.363985 5935 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7cdc5a94-7c68-49f2-95b9-8d23d1355581-metrics-certs\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:30:39.085207 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:39.085182 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d2dc0ad5-ff87549c8-xs57b" Nov 01 20:30:39.085677 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:39.085175 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-d2dc0ad5-ff87549c8-xs57b" event={"ID":"7cdc5a94-7c68-49f2-95b9-8d23d1355581","Type":"ContainerDied","Data":"3ddf4426a0bdf9657bebb48c021481e190f7dacb5f7eddfefa7462df33e2bee8"} Nov 01 20:30:39.085821 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:39.085720 5935 scope.go:122] "RemoveContainer" containerID="ca9fede5f97956aa16b8ff2e5d5f554da676983120099d42db6d91dc25846b5f" Nov 01 20:30:39.112223 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:39.112182 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d2dc0ad5-ff87549c8-xs57b"] Nov 01 20:30:39.137184 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:39.137144 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-d2dc0ad5-ff87549c8-xs57b"] Nov 01 20:30:40.354485 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:30:40.354448 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cdc5a94-7c68-49f2-95b9-8d23d1355581" path="/var/lib/kubelet/pods/7cdc5a94-7c68-49f2-95b9-8d23d1355581/volumes" Nov 01 20:31:05.844968 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:31:05.844929 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:31:11.159920 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:31:11.159871 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-8b6cd6/sub-test-model-8b6cd6-kserve-6c44c95575-sg7gm"] Nov 01 20:31:26.192951 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:31:26.192909 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-8b6cd6/sub-test-model-8b6cd6-kserve-6c44c95575-sg7gm" podUID="50eb270e-173d-4727-9504-1b69379f7f77" containerName="main" containerID="cri-o://1b245a89336d668a1bd81e88ecdb0579b4a98843928f28d2c12aa068f69f1f3b" gracePeriod=45 Nov 01 20:31:26.405436 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:31:26.405397 5935 generic.go:356] "Generic (PLEG): container finished" podID="50eb270e-173d-4727-9504-1b69379f7f77" containerID="1b245a89336d668a1bd81e88ecdb0579b4a98843928f28d2c12aa068f69f1f3b" exitCode=0 Nov 01 20:31:26.405615 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:31:26.405457 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-8b6cd6/sub-test-model-8b6cd6-kserve-6c44c95575-sg7gm" event={"ID":"50eb270e-173d-4727-9504-1b69379f7f77","Type":"ContainerDied","Data":"1b245a89336d668a1bd81e88ecdb0579b4a98843928f28d2c12aa068f69f1f3b"} Nov 01 20:31:26.459056 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:31:26.458989 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-8b6cd6/sub-test-model-8b6cd6-kserve-6c44c95575-sg7gm" Nov 01 20:31:26.543350 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:31:26.543279 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50eb270e-173d-4727-9504-1b69379f7f77-home\" (UniqueName: \"kubernetes.io/empty-dir/50eb270e-173d-4727-9504-1b69379f7f77-home\") pod \"50eb270e-173d-4727-9504-1b69379f7f77\" (UID: \"50eb270e-173d-4727-9504-1b69379f7f77\") " Nov 01 20:31:26.543528 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:31:26.543370 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50eb270e-173d-4727-9504-1b69379f7f77-dshm\" (UniqueName: \"kubernetes.io/empty-dir/50eb270e-173d-4727-9504-1b69379f7f77-dshm\") pod \"50eb270e-173d-4727-9504-1b69379f7f77\" (UID: \"50eb270e-173d-4727-9504-1b69379f7f77\") " Nov 01 20:31:26.543528 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:31:26.543406 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50eb270e-173d-4727-9504-1b69379f7f77-tls-certs\" (UniqueName: \"kubernetes.io/secret/50eb270e-173d-4727-9504-1b69379f7f77-tls-certs\") pod \"50eb270e-173d-4727-9504-1b69379f7f77\" (UID: \"50eb270e-173d-4727-9504-1b69379f7f77\") " Nov 01 20:31:26.543528 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:31:26.543453 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50eb270e-173d-4727-9504-1b69379f7f77-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50eb270e-173d-4727-9504-1b69379f7f77-tmp-dir\") pod \"50eb270e-173d-4727-9504-1b69379f7f77\" (UID: \"50eb270e-173d-4727-9504-1b69379f7f77\") " Nov 01 20:31:26.543528 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:31:26.543502 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50eb270e-173d-4727-9504-1b69379f7f77-kube-api-access-48tjm\" (UniqueName: \"kubernetes.io/projected/50eb270e-173d-4727-9504-1b69379f7f77-kube-api-access-48tjm\") pod \"50eb270e-173d-4727-9504-1b69379f7f77\" (UID: \"50eb270e-173d-4727-9504-1b69379f7f77\") " Nov 01 20:31:26.543733 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:31:26.543550 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50eb270e-173d-4727-9504-1b69379f7f77-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50eb270e-173d-4727-9504-1b69379f7f77-model-cache\") pod \"50eb270e-173d-4727-9504-1b69379f7f77\" (UID: \"50eb270e-173d-4727-9504-1b69379f7f77\") " Nov 01 20:31:26.543733 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:31:26.543576 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50eb270e-173d-4727-9504-1b69379f7f77-home" pod "50eb270e-173d-4727-9504-1b69379f7f77" (UID: "50eb270e-173d-4727-9504-1b69379f7f77"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:31:26.543733 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:31:26.543711 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50eb270e-173d-4727-9504-1b69379f7f77-tmp-dir" pod "50eb270e-173d-4727-9504-1b69379f7f77" (UID: "50eb270e-173d-4727-9504-1b69379f7f77"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:31:26.543973 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:31:26.543941 5935 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/50eb270e-173d-4727-9504-1b69379f7f77-home\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:31:26.543973 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:31:26.543963 5935 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/50eb270e-173d-4727-9504-1b69379f7f77-tmp-dir\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:31:26.544127 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:31:26.544009 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50eb270e-173d-4727-9504-1b69379f7f77-model-cache" pod "50eb270e-173d-4727-9504-1b69379f7f77" (UID: "50eb270e-173d-4727-9504-1b69379f7f77"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:31:26.545874 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:31:26.545847 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50eb270e-173d-4727-9504-1b69379f7f77-kube-api-access-48tjm" pod "50eb270e-173d-4727-9504-1b69379f7f77" (UID: "50eb270e-173d-4727-9504-1b69379f7f77"). InnerVolumeSpecName "kube-api-access-48tjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:31:26.546064 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:31:26.546040 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50eb270e-173d-4727-9504-1b69379f7f77-dshm" pod "50eb270e-173d-4727-9504-1b69379f7f77" (UID: "50eb270e-173d-4727-9504-1b69379f7f77"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:31:26.546064 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:31:26.546054 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50eb270e-173d-4727-9504-1b69379f7f77-tls-certs" pod "50eb270e-173d-4727-9504-1b69379f7f77" (UID: "50eb270e-173d-4727-9504-1b69379f7f77"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:31:26.644926 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:31:26.644885 5935 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/50eb270e-173d-4727-9504-1b69379f7f77-dshm\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:31:26.644926 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:31:26.644918 5935 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50eb270e-173d-4727-9504-1b69379f7f77-tls-certs\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:31:26.644926 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:31:26.644931 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48tjm\" (UniqueName: \"kubernetes.io/projected/50eb270e-173d-4727-9504-1b69379f7f77-kube-api-access-48tjm\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:31:26.645177 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:31:26.644942 5935 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/50eb270e-173d-4727-9504-1b69379f7f77-model-cache\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:31:27.412455 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:31:27.412424 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-8b6cd6/sub-test-model-8b6cd6-kserve-6c44c95575-sg7gm" Nov 01 20:31:27.412956 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:31:27.412461 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-8b6cd6/sub-test-model-8b6cd6-kserve-6c44c95575-sg7gm" event={"ID":"50eb270e-173d-4727-9504-1b69379f7f77","Type":"ContainerDied","Data":"d3bbb0f43af74448e9c6678284745135247743779828964e850f167dd2c8677c"} Nov 01 20:31:27.412956 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:31:27.412520 5935 scope.go:122] "RemoveContainer" containerID="1b245a89336d668a1bd81e88ecdb0579b4a98843928f28d2c12aa068f69f1f3b" Nov 01 20:31:27.434727 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:31:27.434699 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-8b6cd6/sub-test-model-8b6cd6-kserve-6c44c95575-sg7gm"] Nov 01 20:31:27.440727 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:31:27.440698 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-8b6cd6/sub-test-model-8b6cd6-kserve-6c44c95575-sg7gm"] Nov 01 20:31:28.355250 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:31:28.355211 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50eb270e-173d-4727-9504-1b69379f7f77" path="/var/lib/kubelet/pods/50eb270e-173d-4727-9504-1b69379f7f77/volumes" Nov 01 20:31:34.355418 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:31:34.355383 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-3a6480/distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f"] Nov 01 20:31:35.846227 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:31:35.846187 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:31:49.376916 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:31:49.376874 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-3a6480/distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f" podUID="9e1d573b-e388-4d7b-b47a-8a01467c03b2" containerName="main" containerID="cri-o://6538548ecbbd4be1dc654fd71c83800b988ee6fb901db9c0a39e409e4b57843c" gracePeriod=45 Nov 01 20:31:49.484938 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:31:49.484893 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3a6480/distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f" podUID="9e1d573b-e388-4d7b-b47a-8a01467c03b2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": dial tcp 10.133.0.66:8000: connect: connection refused" Nov 01 20:31:49.550853 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:31:49.550815 5935 generic.go:356] "Generic (PLEG): container finished" podID="9e1d573b-e388-4d7b-b47a-8a01467c03b2" containerID="6538548ecbbd4be1dc654fd71c83800b988ee6fb901db9c0a39e409e4b57843c" exitCode=0 Nov 01 20:31:49.551028 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:31:49.550857 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3a6480/distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f" event={"ID":"9e1d573b-e388-4d7b-b47a-8a01467c03b2","Type":"ContainerDied","Data":"6538548ecbbd4be1dc654fd71c83800b988ee6fb901db9c0a39e409e4b57843c"} Nov 01 20:31:49.641269 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:31:49.641204 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3a6480/distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f" Nov 01 20:31:49.814232 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:31:49.814200 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e1d573b-e388-4d7b-b47a-8a01467c03b2-tls-certs\" (UniqueName: \"kubernetes.io/secret/9e1d573b-e388-4d7b-b47a-8a01467c03b2-tls-certs\") pod \"9e1d573b-e388-4d7b-b47a-8a01467c03b2\" (UID: \"9e1d573b-e388-4d7b-b47a-8a01467c03b2\") " Nov 01 20:31:49.814464 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:31:49.814273 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e1d573b-e388-4d7b-b47a-8a01467c03b2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e1d573b-e388-4d7b-b47a-8a01467c03b2-model-cache\") pod \"9e1d573b-e388-4d7b-b47a-8a01467c03b2\" (UID: \"9e1d573b-e388-4d7b-b47a-8a01467c03b2\") " Nov 01 20:31:49.814538 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:31:49.814464 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e1d573b-e388-4d7b-b47a-8a01467c03b2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e1d573b-e388-4d7b-b47a-8a01467c03b2-dshm\") pod \"9e1d573b-e388-4d7b-b47a-8a01467c03b2\" (UID: \"9e1d573b-e388-4d7b-b47a-8a01467c03b2\") " Nov 01 20:31:49.814538 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:31:49.814500 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e1d573b-e388-4d7b-b47a-8a01467c03b2-home\" (UniqueName: \"kubernetes.io/empty-dir/9e1d573b-e388-4d7b-b47a-8a01467c03b2-home\") pod \"9e1d573b-e388-4d7b-b47a-8a01467c03b2\" (UID: \"9e1d573b-e388-4d7b-b47a-8a01467c03b2\") " Nov 01 20:31:49.815887 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:31:49.814560 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e1d573b-e388-4d7b-b47a-8a01467c03b2-kube-api-access-64tft\" (UniqueName: \"kubernetes.io/projected/9e1d573b-e388-4d7b-b47a-8a01467c03b2-kube-api-access-64tft\") pod \"9e1d573b-e388-4d7b-b47a-8a01467c03b2\" (UID: \"9e1d573b-e388-4d7b-b47a-8a01467c03b2\") " Nov 01 20:31:49.815887 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:31:49.814600 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e1d573b-e388-4d7b-b47a-8a01467c03b2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e1d573b-e388-4d7b-b47a-8a01467c03b2-tmp-dir\") pod \"9e1d573b-e388-4d7b-b47a-8a01467c03b2\" (UID: \"9e1d573b-e388-4d7b-b47a-8a01467c03b2\") " Nov 01 20:31:49.815887 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:31:49.814686 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e1d573b-e388-4d7b-b47a-8a01467c03b2-model-cache" pod "9e1d573b-e388-4d7b-b47a-8a01467c03b2" (UID: "9e1d573b-e388-4d7b-b47a-8a01467c03b2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:31:49.815887 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:31:49.815093 5935 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e1d573b-e388-4d7b-b47a-8a01467c03b2-model-cache\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:31:49.815887 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:31:49.815836 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e1d573b-e388-4d7b-b47a-8a01467c03b2-tmp-dir" pod "9e1d573b-e388-4d7b-b47a-8a01467c03b2" (UID: "9e1d573b-e388-4d7b-b47a-8a01467c03b2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:31:49.816295 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:31:49.815941 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e1d573b-e388-4d7b-b47a-8a01467c03b2-home" pod "9e1d573b-e388-4d7b-b47a-8a01467c03b2" (UID: "9e1d573b-e388-4d7b-b47a-8a01467c03b2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:31:49.817715 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:31:49.817683 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e1d573b-e388-4d7b-b47a-8a01467c03b2-tls-certs" pod "9e1d573b-e388-4d7b-b47a-8a01467c03b2" (UID: "9e1d573b-e388-4d7b-b47a-8a01467c03b2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:31:49.817860 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:31:49.817781 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e1d573b-e388-4d7b-b47a-8a01467c03b2-kube-api-access-64tft" pod "9e1d573b-e388-4d7b-b47a-8a01467c03b2" (UID: "9e1d573b-e388-4d7b-b47a-8a01467c03b2"). InnerVolumeSpecName "kube-api-access-64tft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:31:49.818022 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:31:49.818005 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e1d573b-e388-4d7b-b47a-8a01467c03b2-dshm" pod "9e1d573b-e388-4d7b-b47a-8a01467c03b2" (UID: "9e1d573b-e388-4d7b-b47a-8a01467c03b2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:31:49.916680 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:31:49.916592 5935 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e1d573b-e388-4d7b-b47a-8a01467c03b2-tls-certs\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:31:49.916680 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:31:49.916632 5935 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e1d573b-e388-4d7b-b47a-8a01467c03b2-dshm\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:31:49.916680 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:31:49.916645 5935 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e1d573b-e388-4d7b-b47a-8a01467c03b2-home\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:31:49.916680 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:31:49.916658 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64tft\" (UniqueName: \"kubernetes.io/projected/9e1d573b-e388-4d7b-b47a-8a01467c03b2-kube-api-access-64tft\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:31:49.916680 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:31:49.916673 5935 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e1d573b-e388-4d7b-b47a-8a01467c03b2-tmp-dir\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:31:50.386830 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:31:50.386789 5935 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e1d573b_e388_4d7b_b47a_8a01467c03b2.slice\": RecentStats: unable to find data in memory cache]" Nov 01 20:31:50.387376 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:31:50.386965 5935 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e1d573b_e388_4d7b_b47a_8a01467c03b2.slice/crio-534009b076a4ad0d2e4adb11fc1b1e340a3a726625288263ae6dc4fe68e293b3\": RecentStats: unable to find data in memory cache]" Nov 01 20:31:50.558403 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:31:50.558371 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3a6480/distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f" Nov 01 20:31:50.558576 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:31:50.558396 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3a6480/distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f" event={"ID":"9e1d573b-e388-4d7b-b47a-8a01467c03b2","Type":"ContainerDied","Data":"534009b076a4ad0d2e4adb11fc1b1e340a3a726625288263ae6dc4fe68e293b3"} Nov 01 20:31:50.558576 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:31:50.558448 5935 scope.go:122] "RemoveContainer" containerID="6538548ecbbd4be1dc654fd71c83800b988ee6fb901db9c0a39e409e4b57843c" Nov 01 20:31:50.578204 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:31:50.578170 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-3a6480/distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f"] Nov 01 20:31:50.581624 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:31:50.581596 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-3a6480/distinct-w0-3a6480-kserve-5c8ffd5d87-gd75f"] Nov 01 20:31:52.353824 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:31:52.353785 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e1d573b-e388-4d7b-b47a-8a01467c03b2" path="/var/lib/kubelet/pods/9e1d573b-e388-4d7b-b47a-8a01467c03b2/volumes" Nov 01 20:32:00.108963 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:00.108924 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw"] Nov 01 20:32:00.109404 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:00.109175 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw" podUID="ab7a8222-c059-4ecd-8187-fb711ac219c4" containerName="istio-proxy" containerID="cri-o://088a4b4d9b3a3b0e63bd1eda3077fc56d3011c3e660c3611f4dbedee35a0e1bb" gracePeriod=30 Nov 01 20:32:01.344320 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:01.344261 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8b6cd6-85cd59544b-dm2wk"] Nov 01 20:32:01.344739 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:01.344533 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8b6cd6-85cd59544b-dm2wk" podUID="56d6fb4f-c372-490f-9dec-5b36a2ab21e5" containerName="maas-api" containerID="cri-o://f5044c1bb2893658231a8f84cb20a7291bfcce2ac2aa001a923632caee0839a9" gracePeriod=30 Nov 01 20:32:01.619903 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:01.619870 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8b6cd6-85cd59544b-dm2wk" Nov 01 20:32:01.628434 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:01.628400 5935 generic.go:356] "Generic (PLEG): container finished" podID="56d6fb4f-c372-490f-9dec-5b36a2ab21e5" containerID="f5044c1bb2893658231a8f84cb20a7291bfcce2ac2aa001a923632caee0839a9" exitCode=0 Nov 01 20:32:01.628612 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:01.628448 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8b6cd6-85cd59544b-dm2wk" Nov 01 20:32:01.628612 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:01.628482 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8b6cd6-85cd59544b-dm2wk" event={"ID":"56d6fb4f-c372-490f-9dec-5b36a2ab21e5","Type":"ContainerDied","Data":"f5044c1bb2893658231a8f84cb20a7291bfcce2ac2aa001a923632caee0839a9"} Nov 01 20:32:01.628612 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:01.628535 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8b6cd6-85cd59544b-dm2wk" event={"ID":"56d6fb4f-c372-490f-9dec-5b36a2ab21e5","Type":"ContainerDied","Data":"95da55ab40226bec5bbae16d041b6875c5819222757c11315605ab4531b0b5a9"} Nov 01 20:32:01.628612 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:01.628557 5935 scope.go:122] "RemoveContainer" containerID="f5044c1bb2893658231a8f84cb20a7291bfcce2ac2aa001a923632caee0839a9" Nov 01 20:32:01.639272 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:01.639243 5935 scope.go:122] "RemoveContainer" containerID="f5044c1bb2893658231a8f84cb20a7291bfcce2ac2aa001a923632caee0839a9" Nov 01 20:32:01.639693 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:32:01.639665 5935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5044c1bb2893658231a8f84cb20a7291bfcce2ac2aa001a923632caee0839a9\": container with ID starting with f5044c1bb2893658231a8f84cb20a7291bfcce2ac2aa001a923632caee0839a9 not found: ID does not exist" containerID="f5044c1bb2893658231a8f84cb20a7291bfcce2ac2aa001a923632caee0839a9" Nov 01 20:32:01.639810 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:01.639698 5935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5044c1bb2893658231a8f84cb20a7291bfcce2ac2aa001a923632caee0839a9"} err="failed to get container status \"f5044c1bb2893658231a8f84cb20a7291bfcce2ac2aa001a923632caee0839a9\": rpc error: code = NotFound desc = could not find container \"f5044c1bb2893658231a8f84cb20a7291bfcce2ac2aa001a923632caee0839a9\": container with ID starting with f5044c1bb2893658231a8f84cb20a7291bfcce2ac2aa001a923632caee0839a9 not found: ID does not exist" Nov 01 20:32:01.750727 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:01.750676 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56d6fb4f-c372-490f-9dec-5b36a2ab21e5-kube-api-access-qh8z9\" (UniqueName: \"kubernetes.io/projected/56d6fb4f-c372-490f-9dec-5b36a2ab21e5-kube-api-access-qh8z9\") pod \"56d6fb4f-c372-490f-9dec-5b36a2ab21e5\" (UID: \"56d6fb4f-c372-490f-9dec-5b36a2ab21e5\") " Nov 01 20:32:01.751083 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:01.750915 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56d6fb4f-c372-490f-9dec-5b36a2ab21e5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56d6fb4f-c372-490f-9dec-5b36a2ab21e5-maas-api-tls\") pod \"56d6fb4f-c372-490f-9dec-5b36a2ab21e5\" (UID: \"56d6fb4f-c372-490f-9dec-5b36a2ab21e5\") " Nov 01 20:32:01.751178 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:01.751111 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56d6fb4f-c372-490f-9dec-5b36a2ab21e5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56d6fb4f-c372-490f-9dec-5b36a2ab21e5-openshift-service-ca\") pod \"56d6fb4f-c372-490f-9dec-5b36a2ab21e5\" (UID: \"56d6fb4f-c372-490f-9dec-5b36a2ab21e5\") " Nov 01 20:32:01.751229 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:01.751193 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56d6fb4f-c372-490f-9dec-5b36a2ab21e5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/56d6fb4f-c372-490f-9dec-5b36a2ab21e5-metrics-certs\") pod \"56d6fb4f-c372-490f-9dec-5b36a2ab21e5\" (UID: \"56d6fb4f-c372-490f-9dec-5b36a2ab21e5\") " Nov 01 20:32:01.751510 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:01.751480 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56d6fb4f-c372-490f-9dec-5b36a2ab21e5-openshift-service-ca" pod "56d6fb4f-c372-490f-9dec-5b36a2ab21e5" (UID: "56d6fb4f-c372-490f-9dec-5b36a2ab21e5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:32:01.751666 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:01.751645 5935 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/56d6fb4f-c372-490f-9dec-5b36a2ab21e5-openshift-service-ca\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:32:01.753141 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:01.753113 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56d6fb4f-c372-490f-9dec-5b36a2ab21e5-maas-api-tls" pod "56d6fb4f-c372-490f-9dec-5b36a2ab21e5" (UID: "56d6fb4f-c372-490f-9dec-5b36a2ab21e5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:32:01.753141 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:01.753122 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56d6fb4f-c372-490f-9dec-5b36a2ab21e5-kube-api-access-qh8z9" pod "56d6fb4f-c372-490f-9dec-5b36a2ab21e5" (UID: "56d6fb4f-c372-490f-9dec-5b36a2ab21e5"). InnerVolumeSpecName "kube-api-access-qh8z9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:32:01.753268 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:01.753197 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56d6fb4f-c372-490f-9dec-5b36a2ab21e5-metrics-certs" pod "56d6fb4f-c372-490f-9dec-5b36a2ab21e5" (UID: "56d6fb4f-c372-490f-9dec-5b36a2ab21e5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:32:01.853211 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:01.853120 5935 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/56d6fb4f-c372-490f-9dec-5b36a2ab21e5-maas-api-tls\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:32:01.853211 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:01.853158 5935 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/56d6fb4f-c372-490f-9dec-5b36a2ab21e5-metrics-certs\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:32:01.853211 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:01.853170 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qh8z9\" (UniqueName: \"kubernetes.io/projected/56d6fb4f-c372-490f-9dec-5b36a2ab21e5-kube-api-access-qh8z9\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:32:01.952063 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:01.952025 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8b6cd6-85cd59544b-dm2wk"] Nov 01 20:32:01.954357 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:01.954324 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8b6cd6-85cd59544b-dm2wk"] Nov 01 20:32:02.353430 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:02.353390 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56d6fb4f-c372-490f-9dec-5b36a2ab21e5" path="/var/lib/kubelet/pods/56d6fb4f-c372-490f-9dec-5b36a2ab21e5/volumes" Nov 01 20:32:05.374254 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:05.374183 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw" Nov 01 20:32:05.494932 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:05.494896 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab7a8222-c059-4ecd-8187-fb711ac219c4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab7a8222-c059-4ecd-8187-fb711ac219c4-workload-socket\") pod \"ab7a8222-c059-4ecd-8187-fb711ac219c4\" (UID: \"ab7a8222-c059-4ecd-8187-fb711ac219c4\") " Nov 01 20:32:05.495140 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:05.494965 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab7a8222-c059-4ecd-8187-fb711ac219c4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab7a8222-c059-4ecd-8187-fb711ac219c4-istiod-ca-cert\") pod \"ab7a8222-c059-4ecd-8187-fb711ac219c4\" (UID: \"ab7a8222-c059-4ecd-8187-fb711ac219c4\") " Nov 01 20:32:05.495140 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:05.495020 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab7a8222-c059-4ecd-8187-fb711ac219c4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab7a8222-c059-4ecd-8187-fb711ac219c4-workload-certs\") pod \"ab7a8222-c059-4ecd-8187-fb711ac219c4\" (UID: \"ab7a8222-c059-4ecd-8187-fb711ac219c4\") " Nov 01 20:32:05.495140 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:05.495055 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab7a8222-c059-4ecd-8187-fb711ac219c4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab7a8222-c059-4ecd-8187-fb711ac219c4-credential-socket\") pod \"ab7a8222-c059-4ecd-8187-fb711ac219c4\" (UID: \"ab7a8222-c059-4ecd-8187-fb711ac219c4\") " Nov 01 20:32:05.495140 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:05.495103 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab7a8222-c059-4ecd-8187-fb711ac219c4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab7a8222-c059-4ecd-8187-fb711ac219c4-istio-data\") pod \"ab7a8222-c059-4ecd-8187-fb711ac219c4\" (UID: \"ab7a8222-c059-4ecd-8187-fb711ac219c4\") " Nov 01 20:32:05.495386 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:05.495208 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab7a8222-c059-4ecd-8187-fb711ac219c4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab7a8222-c059-4ecd-8187-fb711ac219c4-istio-envoy\") pod \"ab7a8222-c059-4ecd-8187-fb711ac219c4\" (UID: \"ab7a8222-c059-4ecd-8187-fb711ac219c4\") " Nov 01 20:32:05.495386 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:05.495243 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ab7a8222-c059-4ecd-8187-fb711ac219c4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab7a8222-c059-4ecd-8187-fb711ac219c4-istio-podinfo\") pod \"ab7a8222-c059-4ecd-8187-fb711ac219c4\" (UID: \"ab7a8222-c059-4ecd-8187-fb711ac219c4\") " Nov 01 20:32:05.495386 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:05.495243 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab7a8222-c059-4ecd-8187-fb711ac219c4-workload-socket" pod "ab7a8222-c059-4ecd-8187-fb711ac219c4" (UID: "ab7a8222-c059-4ecd-8187-fb711ac219c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:32:05.495386 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:05.495272 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab7a8222-c059-4ecd-8187-fb711ac219c4-istio-token\" (UniqueName: \"kubernetes.io/projected/ab7a8222-c059-4ecd-8187-fb711ac219c4-istio-token\") pod \"ab7a8222-c059-4ecd-8187-fb711ac219c4\" (UID: \"ab7a8222-c059-4ecd-8187-fb711ac219c4\") " Nov 01 20:32:05.495386 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:05.495351 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab7a8222-c059-4ecd-8187-fb711ac219c4-workload-certs" pod "ab7a8222-c059-4ecd-8187-fb711ac219c4" (UID: "ab7a8222-c059-4ecd-8187-fb711ac219c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:32:05.495676 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:05.495424 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab7a8222-c059-4ecd-8187-fb711ac219c4-kube-api-access-bm587\" (UniqueName: \"kubernetes.io/projected/ab7a8222-c059-4ecd-8187-fb711ac219c4-kube-api-access-bm587\") pod \"ab7a8222-c059-4ecd-8187-fb711ac219c4\" (UID: \"ab7a8222-c059-4ecd-8187-fb711ac219c4\") " Nov 01 20:32:05.495676 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:05.495484 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab7a8222-c059-4ecd-8187-fb711ac219c4-istiod-ca-cert" pod "ab7a8222-c059-4ecd-8187-fb711ac219c4" (UID: "ab7a8222-c059-4ecd-8187-fb711ac219c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:32:05.495676 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:05.495493 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab7a8222-c059-4ecd-8187-fb711ac219c4-istio-data" pod "ab7a8222-c059-4ecd-8187-fb711ac219c4" (UID: "ab7a8222-c059-4ecd-8187-fb711ac219c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:32:05.495863 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:05.495843 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab7a8222-c059-4ecd-8187-fb711ac219c4-credential-socket" pod "ab7a8222-c059-4ecd-8187-fb711ac219c4" (UID: "ab7a8222-c059-4ecd-8187-fb711ac219c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:32:05.495961 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:05.495941 5935 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab7a8222-c059-4ecd-8187-fb711ac219c4-workload-socket\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:32:05.495961 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:05.495959 5935 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab7a8222-c059-4ecd-8187-fb711ac219c4-istiod-ca-cert\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:32:05.496160 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:05.495968 5935 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab7a8222-c059-4ecd-8187-fb711ac219c4-workload-certs\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:32:05.496160 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:05.495979 5935 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab7a8222-c059-4ecd-8187-fb711ac219c4-credential-socket\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:32:05.496160 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:05.495988 5935 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab7a8222-c059-4ecd-8187-fb711ac219c4-istio-data\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:32:05.497705 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:05.497679 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ab7a8222-c059-4ecd-8187-fb711ac219c4-istio-podinfo" pod "ab7a8222-c059-4ecd-8187-fb711ac219c4" (UID: "ab7a8222-c059-4ecd-8187-fb711ac219c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:32:05.497813 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:05.497754 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab7a8222-c059-4ecd-8187-fb711ac219c4-kube-api-access-bm587" pod "ab7a8222-c059-4ecd-8187-fb711ac219c4" (UID: "ab7a8222-c059-4ecd-8187-fb711ac219c4"). InnerVolumeSpecName "kube-api-access-bm587". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:32:05.497880 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:05.497863 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab7a8222-c059-4ecd-8187-fb711ac219c4-istio-envoy" pod "ab7a8222-c059-4ecd-8187-fb711ac219c4" (UID: "ab7a8222-c059-4ecd-8187-fb711ac219c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:32:05.498039 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:05.498019 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab7a8222-c059-4ecd-8187-fb711ac219c4-istio-token" pod "ab7a8222-c059-4ecd-8187-fb711ac219c4" (UID: "ab7a8222-c059-4ecd-8187-fb711ac219c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:32:05.596807 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:05.596755 5935 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab7a8222-c059-4ecd-8187-fb711ac219c4-istio-envoy\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:32:05.596807 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:05.596801 5935 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab7a8222-c059-4ecd-8187-fb711ac219c4-istio-podinfo\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:32:05.596807 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:05.596811 5935 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab7a8222-c059-4ecd-8187-fb711ac219c4-istio-token\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:32:05.597067 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:05.596841 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bm587\" (UniqueName: \"kubernetes.io/projected/ab7a8222-c059-4ecd-8187-fb711ac219c4-kube-api-access-bm587\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:32:05.654368 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:05.654250 5935 generic.go:356] "Generic (PLEG): container finished" podID="ab7a8222-c059-4ecd-8187-fb711ac219c4" containerID="088a4b4d9b3a3b0e63bd1eda3077fc56d3011c3e660c3611f4dbedee35a0e1bb" exitCode=0 Nov 01 20:32:05.654368 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:05.654322 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw" event={"ID":"ab7a8222-c059-4ecd-8187-fb711ac219c4","Type":"ContainerDied","Data":"088a4b4d9b3a3b0e63bd1eda3077fc56d3011c3e660c3611f4dbedee35a0e1bb"} Nov 01 20:32:05.654368 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:05.654340 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw" Nov 01 20:32:05.654624 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:05.654403 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw" event={"ID":"ab7a8222-c059-4ecd-8187-fb711ac219c4","Type":"ContainerDied","Data":"a31c74e6682159595a4f2b25d76b7df60f53fb4674779f9878b3da1605f932ff"} Nov 01 20:32:05.654624 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:05.654427 5935 scope.go:122] "RemoveContainer" containerID="088a4b4d9b3a3b0e63bd1eda3077fc56d3011c3e660c3611f4dbedee35a0e1bb" Nov 01 20:32:05.664516 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:05.664501 5935 scope.go:122] "RemoveContainer" containerID="088a4b4d9b3a3b0e63bd1eda3077fc56d3011c3e660c3611f4dbedee35a0e1bb" Nov 01 20:32:05.664823 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:32:05.664800 5935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"088a4b4d9b3a3b0e63bd1eda3077fc56d3011c3e660c3611f4dbedee35a0e1bb\": container with ID starting with 088a4b4d9b3a3b0e63bd1eda3077fc56d3011c3e660c3611f4dbedee35a0e1bb not found: ID does not exist" containerID="088a4b4d9b3a3b0e63bd1eda3077fc56d3011c3e660c3611f4dbedee35a0e1bb" Nov 01 20:32:05.664879 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:05.664831 5935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"088a4b4d9b3a3b0e63bd1eda3077fc56d3011c3e660c3611f4dbedee35a0e1bb"} err="failed to get container status \"088a4b4d9b3a3b0e63bd1eda3077fc56d3011c3e660c3611f4dbedee35a0e1bb\": rpc error: code = NotFound desc = could not find container \"088a4b4d9b3a3b0e63bd1eda3077fc56d3011c3e660c3611f4dbedee35a0e1bb\": container with ID starting with 088a4b4d9b3a3b0e63bd1eda3077fc56d3011c3e660c3611f4dbedee35a0e1bb not found: ID does not exist" Nov 01 20:32:05.676631 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:05.676600 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw"] Nov 01 20:32:05.680737 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:05.680713 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-5370bb-openshift-default-6f94db8cc-njccw"] Nov 01 20:32:05.844473 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:32:05.844440 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:32:06.354148 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:06.354110 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab7a8222-c059-4ecd-8187-fb711ac219c4" path="/var/lib/kubelet/pods/ab7a8222-c059-4ecd-8187-fb711ac219c4/volumes" Nov 01 20:32:08.591102 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:08.591066 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr"] Nov 01 20:32:08.591720 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:08.591665 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr" podUID="92a72792-1735-4970-88ac-ca2a50a5b63e" containerName="istio-proxy" containerID="cri-o://e594c15c9b46c1317f91fa1f3eb65b2444087bbd86b4ebae9b2708754ccd6aa2" gracePeriod=30 Nov 01 20:32:12.800769 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:12.800727 5935 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 20:32:12.801192 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:12.800803 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr" podUID="92a72792-1735-4970-88ac-ca2a50a5b63e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 20:32:13.095342 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:13.095236 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp"] Nov 01 20:32:13.095579 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:13.095528 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp" podUID="e2b197fc-6182-4f88-8e11-d631a9d95b37" containerName="istio-proxy" containerID="cri-o://39744e4d2aab8710a4a0b9466ed5a0b95f2168fbc1549dad738a2da1348fde5b" gracePeriod=30 Nov 01 20:32:13.713649 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:13.713607 5935 generic.go:356] "Generic (PLEG): container finished" podID="92a72792-1735-4970-88ac-ca2a50a5b63e" containerID="e594c15c9b46c1317f91fa1f3eb65b2444087bbd86b4ebae9b2708754ccd6aa2" exitCode=0 Nov 01 20:32:13.713811 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:13.713791 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr" event={"ID":"92a72792-1735-4970-88ac-ca2a50a5b63e","Type":"ContainerDied","Data":"e594c15c9b46c1317f91fa1f3eb65b2444087bbd86b4ebae9b2708754ccd6aa2"} Nov 01 20:32:13.872527 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:13.872462 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr" Nov 01 20:32:14.001411 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:14.001374 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/92a72792-1735-4970-88ac-ca2a50a5b63e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92a72792-1735-4970-88ac-ca2a50a5b63e-istio-podinfo\") pod \"92a72792-1735-4970-88ac-ca2a50a5b63e\" (UID: \"92a72792-1735-4970-88ac-ca2a50a5b63e\") " Nov 01 20:32:14.001411 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:14.001413 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92a72792-1735-4970-88ac-ca2a50a5b63e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a72792-1735-4970-88ac-ca2a50a5b63e-credential-socket\") pod \"92a72792-1735-4970-88ac-ca2a50a5b63e\" (UID: \"92a72792-1735-4970-88ac-ca2a50a5b63e\") " Nov 01 20:32:14.001631 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:14.001439 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92a72792-1735-4970-88ac-ca2a50a5b63e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92a72792-1735-4970-88ac-ca2a50a5b63e-istio-data\") pod \"92a72792-1735-4970-88ac-ca2a50a5b63e\" (UID: \"92a72792-1735-4970-88ac-ca2a50a5b63e\") " Nov 01 20:32:14.001631 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:14.001548 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92a72792-1735-4970-88ac-ca2a50a5b63e-istio-token\" (UniqueName: \"kubernetes.io/projected/92a72792-1735-4970-88ac-ca2a50a5b63e-istio-token\") pod \"92a72792-1735-4970-88ac-ca2a50a5b63e\" (UID: \"92a72792-1735-4970-88ac-ca2a50a5b63e\") " Nov 01 20:32:14.001631 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:14.001580 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92a72792-1735-4970-88ac-ca2a50a5b63e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a72792-1735-4970-88ac-ca2a50a5b63e-workload-socket\") pod \"92a72792-1735-4970-88ac-ca2a50a5b63e\" (UID: \"92a72792-1735-4970-88ac-ca2a50a5b63e\") " Nov 01 20:32:14.001799 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:14.001634 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92a72792-1735-4970-88ac-ca2a50a5b63e-kube-api-access-vgfxc\" (UniqueName: \"kubernetes.io/projected/92a72792-1735-4970-88ac-ca2a50a5b63e-kube-api-access-vgfxc\") pod \"92a72792-1735-4970-88ac-ca2a50a5b63e\" (UID: \"92a72792-1735-4970-88ac-ca2a50a5b63e\") " Nov 01 20:32:14.001799 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:14.001671 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92a72792-1735-4970-88ac-ca2a50a5b63e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92a72792-1735-4970-88ac-ca2a50a5b63e-workload-certs\") pod \"92a72792-1735-4970-88ac-ca2a50a5b63e\" (UID: \"92a72792-1735-4970-88ac-ca2a50a5b63e\") " Nov 01 20:32:14.001799 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:14.001773 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92a72792-1735-4970-88ac-ca2a50a5b63e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92a72792-1735-4970-88ac-ca2a50a5b63e-istiod-ca-cert\") pod \"92a72792-1735-4970-88ac-ca2a50a5b63e\" (UID: \"92a72792-1735-4970-88ac-ca2a50a5b63e\") " Nov 01 20:32:14.001967 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:14.001780 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92a72792-1735-4970-88ac-ca2a50a5b63e-credential-socket" pod "92a72792-1735-4970-88ac-ca2a50a5b63e" (UID: "92a72792-1735-4970-88ac-ca2a50a5b63e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:32:14.001967 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:14.001794 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92a72792-1735-4970-88ac-ca2a50a5b63e-istio-data" pod "92a72792-1735-4970-88ac-ca2a50a5b63e" (UID: "92a72792-1735-4970-88ac-ca2a50a5b63e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:32:14.001967 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:14.001858 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92a72792-1735-4970-88ac-ca2a50a5b63e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92a72792-1735-4970-88ac-ca2a50a5b63e-istio-envoy\") pod \"92a72792-1735-4970-88ac-ca2a50a5b63e\" (UID: \"92a72792-1735-4970-88ac-ca2a50a5b63e\") " Nov 01 20:32:14.001967 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:14.001865 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92a72792-1735-4970-88ac-ca2a50a5b63e-workload-socket" pod "92a72792-1735-4970-88ac-ca2a50a5b63e" (UID: "92a72792-1735-4970-88ac-ca2a50a5b63e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:32:14.002182 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:14.002009 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92a72792-1735-4970-88ac-ca2a50a5b63e-workload-certs" pod "92a72792-1735-4970-88ac-ca2a50a5b63e" (UID: "92a72792-1735-4970-88ac-ca2a50a5b63e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:32:14.002243 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:14.002181 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92a72792-1735-4970-88ac-ca2a50a5b63e-istiod-ca-cert" pod "92a72792-1735-4970-88ac-ca2a50a5b63e" (UID: "92a72792-1735-4970-88ac-ca2a50a5b63e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:32:14.002420 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:14.002397 5935 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a72792-1735-4970-88ac-ca2a50a5b63e-workload-socket\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:32:14.002420 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:14.002421 5935 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92a72792-1735-4970-88ac-ca2a50a5b63e-workload-certs\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:32:14.002597 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:14.002435 5935 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92a72792-1735-4970-88ac-ca2a50a5b63e-istiod-ca-cert\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:32:14.002597 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:14.002448 5935 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a72792-1735-4970-88ac-ca2a50a5b63e-credential-socket\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:32:14.002597 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:14.002461 5935 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92a72792-1735-4970-88ac-ca2a50a5b63e-istio-data\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:32:14.003787 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:14.003768 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92a72792-1735-4970-88ac-ca2a50a5b63e-istio-token" pod "92a72792-1735-4970-88ac-ca2a50a5b63e" (UID: "92a72792-1735-4970-88ac-ca2a50a5b63e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:32:14.003899 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:14.003884 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92a72792-1735-4970-88ac-ca2a50a5b63e-kube-api-access-vgfxc" pod "92a72792-1735-4970-88ac-ca2a50a5b63e" (UID: "92a72792-1735-4970-88ac-ca2a50a5b63e"). InnerVolumeSpecName "kube-api-access-vgfxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:32:14.004093 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:14.004073 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/92a72792-1735-4970-88ac-ca2a50a5b63e-istio-podinfo" pod "92a72792-1735-4970-88ac-ca2a50a5b63e" (UID: "92a72792-1735-4970-88ac-ca2a50a5b63e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:32:14.004181 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:14.004124 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92a72792-1735-4970-88ac-ca2a50a5b63e-istio-envoy" pod "92a72792-1735-4970-88ac-ca2a50a5b63e" (UID: "92a72792-1735-4970-88ac-ca2a50a5b63e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:32:14.104148 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:14.104105 5935 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92a72792-1735-4970-88ac-ca2a50a5b63e-istio-envoy\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:32:14.104148 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:14.104146 5935 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92a72792-1735-4970-88ac-ca2a50a5b63e-istio-podinfo\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:32:14.104387 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:14.104159 5935 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92a72792-1735-4970-88ac-ca2a50a5b63e-istio-token\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:32:14.104387 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:14.104172 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgfxc\" (UniqueName: \"kubernetes.io/projected/92a72792-1735-4970-88ac-ca2a50a5b63e-kube-api-access-vgfxc\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:32:14.721813 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:14.721782 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr" Nov 01 20:32:14.722008 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:14.721786 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr" event={"ID":"92a72792-1735-4970-88ac-ca2a50a5b63e","Type":"ContainerDied","Data":"fe104f2f2528f08751fa1f4668202b15803a934d51fe4b18368a9220111a35b7"} Nov 01 20:32:14.722008 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:14.721925 5935 scope.go:122] "RemoveContainer" containerID="e594c15c9b46c1317f91fa1f3eb65b2444087bbd86b4ebae9b2708754ccd6aa2" Nov 01 20:32:14.743055 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:14.743013 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr"] Nov 01 20:32:14.745270 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:14.745246 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8b6cd6-openshift-default-744b88677-ncdrr"] Nov 01 20:32:15.420576 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:15.420491 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-036c69-79cd8c8b6f-fxpb5"] Nov 01 20:32:16.353296 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:16.353260 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92a72792-1735-4970-88ac-ca2a50a5b63e" path="/var/lib/kubelet/pods/92a72792-1735-4970-88ac-ca2a50a5b63e/volumes" Nov 01 20:32:18.363892 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:18.363864 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp" Nov 01 20:32:18.553054 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:18.553022 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2b197fc-6182-4f88-8e11-d631a9d95b37-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2b197fc-6182-4f88-8e11-d631a9d95b37-workload-certs\") pod \"e2b197fc-6182-4f88-8e11-d631a9d95b37\" (UID: \"e2b197fc-6182-4f88-8e11-d631a9d95b37\") " Nov 01 20:32:18.553212 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:18.553070 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2b197fc-6182-4f88-8e11-d631a9d95b37-istio-token\" (UniqueName: \"kubernetes.io/projected/e2b197fc-6182-4f88-8e11-d631a9d95b37-istio-token\") pod \"e2b197fc-6182-4f88-8e11-d631a9d95b37\" (UID: \"e2b197fc-6182-4f88-8e11-d631a9d95b37\") " Nov 01 20:32:18.553212 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:18.553104 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2b197fc-6182-4f88-8e11-d631a9d95b37-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2b197fc-6182-4f88-8e11-d631a9d95b37-workload-socket\") pod \"e2b197fc-6182-4f88-8e11-d631a9d95b37\" (UID: \"e2b197fc-6182-4f88-8e11-d631a9d95b37\") " Nov 01 20:32:18.553212 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:18.553146 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e2b197fc-6182-4f88-8e11-d631a9d95b37-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2b197fc-6182-4f88-8e11-d631a9d95b37-istio-podinfo\") pod \"e2b197fc-6182-4f88-8e11-d631a9d95b37\" (UID: \"e2b197fc-6182-4f88-8e11-d631a9d95b37\") " Nov 01 20:32:18.553212 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:18.553192 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2b197fc-6182-4f88-8e11-d631a9d95b37-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2b197fc-6182-4f88-8e11-d631a9d95b37-credential-socket\") pod \"e2b197fc-6182-4f88-8e11-d631a9d95b37\" (UID: \"e2b197fc-6182-4f88-8e11-d631a9d95b37\") " Nov 01 20:32:18.553528 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:18.553232 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2b197fc-6182-4f88-8e11-d631a9d95b37-kube-api-access-nhnzz\" (UniqueName: \"kubernetes.io/projected/e2b197fc-6182-4f88-8e11-d631a9d95b37-kube-api-access-nhnzz\") pod \"e2b197fc-6182-4f88-8e11-d631a9d95b37\" (UID: \"e2b197fc-6182-4f88-8e11-d631a9d95b37\") " Nov 01 20:32:18.553528 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:18.553364 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2b197fc-6182-4f88-8e11-d631a9d95b37-workload-certs" pod "e2b197fc-6182-4f88-8e11-d631a9d95b37" (UID: "e2b197fc-6182-4f88-8e11-d631a9d95b37"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:32:18.553528 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:18.553380 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2b197fc-6182-4f88-8e11-d631a9d95b37-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2b197fc-6182-4f88-8e11-d631a9d95b37-istiod-ca-cert\") pod \"e2b197fc-6182-4f88-8e11-d631a9d95b37\" (UID: \"e2b197fc-6182-4f88-8e11-d631a9d95b37\") " Nov 01 20:32:18.553528 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:18.553503 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2b197fc-6182-4f88-8e11-d631a9d95b37-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2b197fc-6182-4f88-8e11-d631a9d95b37-istio-envoy\") pod \"e2b197fc-6182-4f88-8e11-d631a9d95b37\" (UID: \"e2b197fc-6182-4f88-8e11-d631a9d95b37\") " Nov 01 20:32:18.553528 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:18.553506 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2b197fc-6182-4f88-8e11-d631a9d95b37-workload-socket" pod "e2b197fc-6182-4f88-8e11-d631a9d95b37" (UID: "e2b197fc-6182-4f88-8e11-d631a9d95b37"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:32:18.553777 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:18.553537 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2b197fc-6182-4f88-8e11-d631a9d95b37-credential-socket" pod "e2b197fc-6182-4f88-8e11-d631a9d95b37" (UID: "e2b197fc-6182-4f88-8e11-d631a9d95b37"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:32:18.553777 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:18.553542 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2b197fc-6182-4f88-8e11-d631a9d95b37-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2b197fc-6182-4f88-8e11-d631a9d95b37-istio-data\") pod \"e2b197fc-6182-4f88-8e11-d631a9d95b37\" (UID: \"e2b197fc-6182-4f88-8e11-d631a9d95b37\") " Nov 01 20:32:18.553777 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:18.553732 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2b197fc-6182-4f88-8e11-d631a9d95b37-istiod-ca-cert" pod "e2b197fc-6182-4f88-8e11-d631a9d95b37" (UID: "e2b197fc-6182-4f88-8e11-d631a9d95b37"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:32:18.553947 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:18.553899 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2b197fc-6182-4f88-8e11-d631a9d95b37-istio-data" pod "e2b197fc-6182-4f88-8e11-d631a9d95b37" (UID: "e2b197fc-6182-4f88-8e11-d631a9d95b37"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:32:18.554047 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:18.554028 5935 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2b197fc-6182-4f88-8e11-d631a9d95b37-istiod-ca-cert\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:32:18.554114 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:18.554048 5935 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2b197fc-6182-4f88-8e11-d631a9d95b37-istio-data\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:32:18.554114 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:18.554062 5935 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2b197fc-6182-4f88-8e11-d631a9d95b37-workload-certs\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:32:18.554114 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:18.554076 5935 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2b197fc-6182-4f88-8e11-d631a9d95b37-workload-socket\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:32:18.554114 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:18.554090 5935 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2b197fc-6182-4f88-8e11-d631a9d95b37-credential-socket\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:32:18.555542 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:18.555517 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2b197fc-6182-4f88-8e11-d631a9d95b37-kube-api-access-nhnzz" pod "e2b197fc-6182-4f88-8e11-d631a9d95b37" (UID: "e2b197fc-6182-4f88-8e11-d631a9d95b37"). InnerVolumeSpecName "kube-api-access-nhnzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:32:18.555542 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:18.555533 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2b197fc-6182-4f88-8e11-d631a9d95b37-istio-token" pod "e2b197fc-6182-4f88-8e11-d631a9d95b37" (UID: "e2b197fc-6182-4f88-8e11-d631a9d95b37"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:32:18.555912 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:18.555896 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e2b197fc-6182-4f88-8e11-d631a9d95b37-istio-podinfo" pod "e2b197fc-6182-4f88-8e11-d631a9d95b37" (UID: "e2b197fc-6182-4f88-8e11-d631a9d95b37"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:32:18.555984 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:18.555910 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2b197fc-6182-4f88-8e11-d631a9d95b37-istio-envoy" pod "e2b197fc-6182-4f88-8e11-d631a9d95b37" (UID: "e2b197fc-6182-4f88-8e11-d631a9d95b37"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:32:18.654724 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:18.654688 5935 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2b197fc-6182-4f88-8e11-d631a9d95b37-istio-token\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:32:18.654724 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:18.654719 5935 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2b197fc-6182-4f88-8e11-d631a9d95b37-istio-podinfo\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:32:18.654724 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:18.654732 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhnzz\" (UniqueName: \"kubernetes.io/projected/e2b197fc-6182-4f88-8e11-d631a9d95b37-kube-api-access-nhnzz\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:32:18.654976 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:18.654743 5935 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2b197fc-6182-4f88-8e11-d631a9d95b37-istio-envoy\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:32:18.745755 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:18.745721 5935 generic.go:356] "Generic (PLEG): container finished" podID="e2b197fc-6182-4f88-8e11-d631a9d95b37" containerID="39744e4d2aab8710a4a0b9466ed5a0b95f2168fbc1549dad738a2da1348fde5b" exitCode=0 Nov 01 20:32:18.745980 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:18.745785 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp" Nov 01 20:32:18.745980 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:18.745807 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp" event={"ID":"e2b197fc-6182-4f88-8e11-d631a9d95b37","Type":"ContainerDied","Data":"39744e4d2aab8710a4a0b9466ed5a0b95f2168fbc1549dad738a2da1348fde5b"} Nov 01 20:32:18.745980 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:18.745879 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp" event={"ID":"e2b197fc-6182-4f88-8e11-d631a9d95b37","Type":"ContainerDied","Data":"a40d9dc91db3411b28cdd7756486759432539b3940783ef4112976652335015a"} Nov 01 20:32:18.745980 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:18.745910 5935 scope.go:122] "RemoveContainer" containerID="39744e4d2aab8710a4a0b9466ed5a0b95f2168fbc1549dad738a2da1348fde5b" Nov 01 20:32:18.756865 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:18.756843 5935 scope.go:122] "RemoveContainer" containerID="39744e4d2aab8710a4a0b9466ed5a0b95f2168fbc1549dad738a2da1348fde5b" Nov 01 20:32:18.757142 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:32:18.757119 5935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39744e4d2aab8710a4a0b9466ed5a0b95f2168fbc1549dad738a2da1348fde5b\": container with ID starting with 39744e4d2aab8710a4a0b9466ed5a0b95f2168fbc1549dad738a2da1348fde5b not found: ID does not exist" containerID="39744e4d2aab8710a4a0b9466ed5a0b95f2168fbc1549dad738a2da1348fde5b" Nov 01 20:32:18.757183 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:18.757150 5935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39744e4d2aab8710a4a0b9466ed5a0b95f2168fbc1549dad738a2da1348fde5b"} err="failed to get container status \"39744e4d2aab8710a4a0b9466ed5a0b95f2168fbc1549dad738a2da1348fde5b\": rpc error: code = NotFound desc = could not find container \"39744e4d2aab8710a4a0b9466ed5a0b95f2168fbc1549dad738a2da1348fde5b\": container with ID starting with 39744e4d2aab8710a4a0b9466ed5a0b95f2168fbc1549dad738a2da1348fde5b not found: ID does not exist" Nov 01 20:32:18.770151 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:18.770123 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp"] Nov 01 20:32:18.773858 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:18.773832 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-3a6480-openshift-default-cd95ffff7-xbqfp"] Nov 01 20:32:20.353844 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:20.353803 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2b197fc-6182-4f88-8e11-d631a9d95b37" path="/var/lib/kubelet/pods/e2b197fc-6182-4f88-8e11-d631a9d95b37/volumes" Nov 01 20:32:20.420905 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:20.420856 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-036c69-79cd8c8b6f-fxpb5" podUID="624bb371-6040-4cd4-a668-a8afc016d132" containerName="payload-processing" containerID="cri-o://fe2827cfb5d9c4adf9c8cb52d85dcfafbd21e70d88b11f1fa46cd667fe43c447" gracePeriod=25 Nov 01 20:32:22.225231 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:22.225194 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-036c69-79cd8c8b6f-fxpb5" podUID="624bb371-6040-4cd4-a668-a8afc016d132" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 20:32:22.332674 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:22.332638 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd"] Nov 01 20:32:22.332927 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:22.332878 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd" podUID="5cb9e245-7141-402d-9601-6f374e26fff3" containerName="istio-proxy" containerID="cri-o://56a4682aedcc5a3c462dd7638b191dea7f2bb66633345428601b436421a05e13" gracePeriod=30 Nov 01 20:32:27.595262 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:27.595235 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd" Nov 01 20:32:27.654112 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:27.654031 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cb9e245-7141-402d-9601-6f374e26fff3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cb9e245-7141-402d-9601-6f374e26fff3-workload-socket\") pod \"5cb9e245-7141-402d-9601-6f374e26fff3\" (UID: \"5cb9e245-7141-402d-9601-6f374e26fff3\") " Nov 01 20:32:27.654112 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:27.654083 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cb9e245-7141-402d-9601-6f374e26fff3-istio-token\" (UniqueName: \"kubernetes.io/projected/5cb9e245-7141-402d-9601-6f374e26fff3-istio-token\") pod \"5cb9e245-7141-402d-9601-6f374e26fff3\" (UID: \"5cb9e245-7141-402d-9601-6f374e26fff3\") " Nov 01 20:32:27.654112 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:27.654109 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5cb9e245-7141-402d-9601-6f374e26fff3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cb9e245-7141-402d-9601-6f374e26fff3-istio-podinfo\") pod \"5cb9e245-7141-402d-9601-6f374e26fff3\" (UID: \"5cb9e245-7141-402d-9601-6f374e26fff3\") " Nov 01 20:32:27.654442 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:27.654145 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cb9e245-7141-402d-9601-6f374e26fff3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cb9e245-7141-402d-9601-6f374e26fff3-istio-data\") pod \"5cb9e245-7141-402d-9601-6f374e26fff3\" (UID: \"5cb9e245-7141-402d-9601-6f374e26fff3\") " Nov 01 20:32:27.654442 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:27.654276 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cb9e245-7141-402d-9601-6f374e26fff3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cb9e245-7141-402d-9601-6f374e26fff3-istio-envoy\") pod \"5cb9e245-7141-402d-9601-6f374e26fff3\" (UID: \"5cb9e245-7141-402d-9601-6f374e26fff3\") " Nov 01 20:32:27.654442 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:27.654329 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cb9e245-7141-402d-9601-6f374e26fff3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cb9e245-7141-402d-9601-6f374e26fff3-credential-socket\") pod \"5cb9e245-7141-402d-9601-6f374e26fff3\" (UID: \"5cb9e245-7141-402d-9601-6f374e26fff3\") " Nov 01 20:32:27.654442 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:27.654354 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cb9e245-7141-402d-9601-6f374e26fff3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cb9e245-7141-402d-9601-6f374e26fff3-workload-certs\") pod \"5cb9e245-7141-402d-9601-6f374e26fff3\" (UID: \"5cb9e245-7141-402d-9601-6f374e26fff3\") " Nov 01 20:32:27.654442 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:27.654374 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cb9e245-7141-402d-9601-6f374e26fff3-workload-socket" pod "5cb9e245-7141-402d-9601-6f374e26fff3" (UID: "5cb9e245-7141-402d-9601-6f374e26fff3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:32:27.654442 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:27.654430 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cb9e245-7141-402d-9601-6f374e26fff3-kube-api-access-9k6t6\" (UniqueName: \"kubernetes.io/projected/5cb9e245-7141-402d-9601-6f374e26fff3-kube-api-access-9k6t6\") pod \"5cb9e245-7141-402d-9601-6f374e26fff3\" (UID: \"5cb9e245-7141-402d-9601-6f374e26fff3\") " Nov 01 20:32:27.654771 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:27.654458 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cb9e245-7141-402d-9601-6f374e26fff3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cb9e245-7141-402d-9601-6f374e26fff3-istiod-ca-cert\") pod \"5cb9e245-7141-402d-9601-6f374e26fff3\" (UID: \"5cb9e245-7141-402d-9601-6f374e26fff3\") " Nov 01 20:32:27.654771 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:27.654620 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cb9e245-7141-402d-9601-6f374e26fff3-credential-socket" pod "5cb9e245-7141-402d-9601-6f374e26fff3" (UID: "5cb9e245-7141-402d-9601-6f374e26fff3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:32:27.654912 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:27.654890 5935 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cb9e245-7141-402d-9601-6f374e26fff3-credential-socket\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:32:27.654984 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:27.654906 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cb9e245-7141-402d-9601-6f374e26fff3-workload-certs" pod "5cb9e245-7141-402d-9601-6f374e26fff3" (UID: "5cb9e245-7141-402d-9601-6f374e26fff3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:32:27.654984 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:27.654918 5935 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cb9e245-7141-402d-9601-6f374e26fff3-workload-socket\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:32:27.655125 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:27.655100 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cb9e245-7141-402d-9601-6f374e26fff3-istiod-ca-cert" pod "5cb9e245-7141-402d-9601-6f374e26fff3" (UID: "5cb9e245-7141-402d-9601-6f374e26fff3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:32:27.655195 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:27.655168 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cb9e245-7141-402d-9601-6f374e26fff3-istio-data" pod "5cb9e245-7141-402d-9601-6f374e26fff3" (UID: "5cb9e245-7141-402d-9601-6f374e26fff3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:32:27.656566 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:27.656536 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cb9e245-7141-402d-9601-6f374e26fff3-istio-envoy" pod "5cb9e245-7141-402d-9601-6f374e26fff3" (UID: "5cb9e245-7141-402d-9601-6f374e26fff3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:32:27.656566 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:27.656541 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5cb9e245-7141-402d-9601-6f374e26fff3-istio-podinfo" pod "5cb9e245-7141-402d-9601-6f374e26fff3" (UID: "5cb9e245-7141-402d-9601-6f374e26fff3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:32:27.656730 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:27.656580 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cb9e245-7141-402d-9601-6f374e26fff3-istio-token" pod "5cb9e245-7141-402d-9601-6f374e26fff3" (UID: "5cb9e245-7141-402d-9601-6f374e26fff3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:32:27.656769 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:27.656750 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cb9e245-7141-402d-9601-6f374e26fff3-kube-api-access-9k6t6" pod "5cb9e245-7141-402d-9601-6f374e26fff3" (UID: "5cb9e245-7141-402d-9601-6f374e26fff3"). InnerVolumeSpecName "kube-api-access-9k6t6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:32:27.756230 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:27.756189 5935 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cb9e245-7141-402d-9601-6f374e26fff3-istio-envoy\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:32:27.756230 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:27.756223 5935 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cb9e245-7141-402d-9601-6f374e26fff3-workload-certs\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:32:27.756230 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:27.756238 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9k6t6\" (UniqueName: \"kubernetes.io/projected/5cb9e245-7141-402d-9601-6f374e26fff3-kube-api-access-9k6t6\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:32:27.756489 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:27.756252 5935 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cb9e245-7141-402d-9601-6f374e26fff3-istiod-ca-cert\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:32:27.756489 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:27.756261 5935 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cb9e245-7141-402d-9601-6f374e26fff3-istio-token\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:32:27.756489 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:27.756269 5935 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cb9e245-7141-402d-9601-6f374e26fff3-istio-podinfo\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:32:27.756489 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:27.756279 5935 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cb9e245-7141-402d-9601-6f374e26fff3-istio-data\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:32:27.802491 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:27.802455 5935 generic.go:356] "Generic (PLEG): container finished" podID="5cb9e245-7141-402d-9601-6f374e26fff3" containerID="56a4682aedcc5a3c462dd7638b191dea7f2bb66633345428601b436421a05e13" exitCode=0 Nov 01 20:32:27.802657 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:27.802508 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd" Nov 01 20:32:27.802657 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:27.802574 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd" event={"ID":"5cb9e245-7141-402d-9601-6f374e26fff3","Type":"ContainerDied","Data":"56a4682aedcc5a3c462dd7638b191dea7f2bb66633345428601b436421a05e13"} Nov 01 20:32:27.802657 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:27.802646 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd" event={"ID":"5cb9e245-7141-402d-9601-6f374e26fff3","Type":"ContainerDied","Data":"3a4c243b3019e051b910e6c94b6d910ffb138a9eee38d33ea13753e8328ffe23"} Nov 01 20:32:27.802777 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:27.802668 5935 scope.go:122] "RemoveContainer" containerID="56a4682aedcc5a3c462dd7638b191dea7f2bb66633345428601b436421a05e13" Nov 01 20:32:27.813078 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:27.813057 5935 scope.go:122] "RemoveContainer" containerID="56a4682aedcc5a3c462dd7638b191dea7f2bb66633345428601b436421a05e13" Nov 01 20:32:27.813421 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:32:27.813399 5935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56a4682aedcc5a3c462dd7638b191dea7f2bb66633345428601b436421a05e13\": container with ID starting with 56a4682aedcc5a3c462dd7638b191dea7f2bb66633345428601b436421a05e13 not found: ID does not exist" containerID="56a4682aedcc5a3c462dd7638b191dea7f2bb66633345428601b436421a05e13" Nov 01 20:32:27.813497 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:27.813438 5935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56a4682aedcc5a3c462dd7638b191dea7f2bb66633345428601b436421a05e13"} err="failed to get container status \"56a4682aedcc5a3c462dd7638b191dea7f2bb66633345428601b436421a05e13\": rpc error: code = NotFound desc = could not find container \"56a4682aedcc5a3c462dd7638b191dea7f2bb66633345428601b436421a05e13\": container with ID starting with 56a4682aedcc5a3c462dd7638b191dea7f2bb66633345428601b436421a05e13 not found: ID does not exist" Nov 01 20:32:27.824789 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:27.824755 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd"] Nov 01 20:32:27.829738 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:27.829713 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-036c69-openshift-default-8d6df548c-67jzd"] Nov 01 20:32:27.887854 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:27.887805 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-a6873b-79b7f97879-9dhts"] Nov 01 20:32:28.353170 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:28.353122 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cb9e245-7141-402d-9601-6f374e26fff3" path="/var/lib/kubelet/pods/5cb9e245-7141-402d-9601-6f374e26fff3/volumes" Nov 01 20:32:28.925719 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:28.925688 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-29b57c-7b87bbfff-2nmvw"] Nov 01 20:32:31.612324 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:31.612271 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-29b57c-69b9ffd5f7-rj47z"] Nov 01 20:32:31.613154 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:31.613092 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-29b57c-69b9ffd5f7-rj47z" podUID="42fe7bf5-319c-449a-aa5a-28b1e2d1f008" containerName="maas-api" containerID="cri-o://08f680832649f004297c84a621796145c4a15a47c9f33341732d2623b209fcdd" gracePeriod=30 Nov 01 20:32:31.836674 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:31.836636 5935 generic.go:356] "Generic (PLEG): container finished" podID="42fe7bf5-319c-449a-aa5a-28b1e2d1f008" containerID="08f680832649f004297c84a621796145c4a15a47c9f33341732d2623b209fcdd" exitCode=0 Nov 01 20:32:31.836906 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:31.836706 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-29b57c-69b9ffd5f7-rj47z" event={"ID":"42fe7bf5-319c-449a-aa5a-28b1e2d1f008","Type":"ContainerDied","Data":"08f680832649f004297c84a621796145c4a15a47c9f33341732d2623b209fcdd"} Nov 01 20:32:31.891759 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:31.891732 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-29b57c-69b9ffd5f7-rj47z" Nov 01 20:32:32.010121 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:32.010085 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42fe7bf5-319c-449a-aa5a-28b1e2d1f008-kube-api-access-dzl4t\" (UniqueName: \"kubernetes.io/projected/42fe7bf5-319c-449a-aa5a-28b1e2d1f008-kube-api-access-dzl4t\") pod \"42fe7bf5-319c-449a-aa5a-28b1e2d1f008\" (UID: \"42fe7bf5-319c-449a-aa5a-28b1e2d1f008\") " Nov 01 20:32:32.010355 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:32.010188 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42fe7bf5-319c-449a-aa5a-28b1e2d1f008-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42fe7bf5-319c-449a-aa5a-28b1e2d1f008-openshift-service-ca\") pod \"42fe7bf5-319c-449a-aa5a-28b1e2d1f008\" (UID: \"42fe7bf5-319c-449a-aa5a-28b1e2d1f008\") " Nov 01 20:32:32.010355 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:32.010262 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42fe7bf5-319c-449a-aa5a-28b1e2d1f008-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42fe7bf5-319c-449a-aa5a-28b1e2d1f008-maas-api-tls\") pod \"42fe7bf5-319c-449a-aa5a-28b1e2d1f008\" (UID: \"42fe7bf5-319c-449a-aa5a-28b1e2d1f008\") " Nov 01 20:32:32.010355 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:32.010280 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42fe7bf5-319c-449a-aa5a-28b1e2d1f008-metrics-certs\" (UniqueName: \"kubernetes.io/secret/42fe7bf5-319c-449a-aa5a-28b1e2d1f008-metrics-certs\") pod \"42fe7bf5-319c-449a-aa5a-28b1e2d1f008\" (UID: \"42fe7bf5-319c-449a-aa5a-28b1e2d1f008\") " Nov 01 20:32:32.010768 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:32.010736 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42fe7bf5-319c-449a-aa5a-28b1e2d1f008-openshift-service-ca" pod "42fe7bf5-319c-449a-aa5a-28b1e2d1f008" (UID: "42fe7bf5-319c-449a-aa5a-28b1e2d1f008"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:32:32.012473 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:32.012448 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42fe7bf5-319c-449a-aa5a-28b1e2d1f008-kube-api-access-dzl4t" pod "42fe7bf5-319c-449a-aa5a-28b1e2d1f008" (UID: "42fe7bf5-319c-449a-aa5a-28b1e2d1f008"). InnerVolumeSpecName "kube-api-access-dzl4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:32:32.012602 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:32.012582 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42fe7bf5-319c-449a-aa5a-28b1e2d1f008-maas-api-tls" pod "42fe7bf5-319c-449a-aa5a-28b1e2d1f008" (UID: "42fe7bf5-319c-449a-aa5a-28b1e2d1f008"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:32:32.012663 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:32.012644 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42fe7bf5-319c-449a-aa5a-28b1e2d1f008-metrics-certs" pod "42fe7bf5-319c-449a-aa5a-28b1e2d1f008" (UID: "42fe7bf5-319c-449a-aa5a-28b1e2d1f008"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:32:32.111394 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:32.111358 5935 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/42fe7bf5-319c-449a-aa5a-28b1e2d1f008-openshift-service-ca\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:32:32.111394 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:32.111388 5935 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/42fe7bf5-319c-449a-aa5a-28b1e2d1f008-maas-api-tls\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:32:32.111394 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:32.111399 5935 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/42fe7bf5-319c-449a-aa5a-28b1e2d1f008-metrics-certs\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:32:32.111663 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:32.111409 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzl4t\" (UniqueName: \"kubernetes.io/projected/42fe7bf5-319c-449a-aa5a-28b1e2d1f008-kube-api-access-dzl4t\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:32:32.225148 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:32.225110 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-036c69-79cd8c8b6f-fxpb5" podUID="624bb371-6040-4cd4-a668-a8afc016d132" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 20:32:32.849642 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:32.849607 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-29b57c-69b9ffd5f7-rj47z" Nov 01 20:32:32.850075 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:32.849606 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-29b57c-69b9ffd5f7-rj47z" event={"ID":"42fe7bf5-319c-449a-aa5a-28b1e2d1f008","Type":"ContainerDied","Data":"56b668f62ca601b84fa2495308f4e5fd0a63b89f233515ab6786bc560c59c067"} Nov 01 20:32:32.850075 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:32.849769 5935 scope.go:122] "RemoveContainer" containerID="08f680832649f004297c84a621796145c4a15a47c9f33341732d2623b209fcdd" Nov 01 20:32:32.867052 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:32.867024 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-29b57c-69b9ffd5f7-rj47z"] Nov 01 20:32:32.869032 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:32.869009 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-29b57c-69b9ffd5f7-rj47z"] Nov 01 20:32:32.889126 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:32.889089 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-a6873b-79b7f97879-9dhts" podUID="d20477dc-2e11-4469-879d-eb1629c24898" containerName="payload-processing" containerID="cri-o://8e27bcd5732252c3ce733f5c6b01c9aa5989465f0c7eb4b8ef8ff77685711bcc" gracePeriod=25 Nov 01 20:32:33.151608 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:33.151573 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-a6873b-79b7f97879-9dhts" Nov 01 20:32:33.222128 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:33.222098 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d20477dc-2e11-4469-879d-eb1629c24898-kube-api-access-n9zfv\" (UniqueName: \"kubernetes.io/projected/d20477dc-2e11-4469-879d-eb1629c24898-kube-api-access-n9zfv\") pod \"d20477dc-2e11-4469-879d-eb1629c24898\" (UID: \"d20477dc-2e11-4469-879d-eb1629c24898\") " Nov 01 20:32:33.222369 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:33.222169 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d20477dc-2e11-4469-879d-eb1629c24898-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d20477dc-2e11-4469-879d-eb1629c24898-plugins-config-volume\") pod \"d20477dc-2e11-4469-879d-eb1629c24898\" (UID: \"d20477dc-2e11-4469-879d-eb1629c24898\") " Nov 01 20:32:33.222583 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:33.222563 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d20477dc-2e11-4469-879d-eb1629c24898-plugins-config-volume" pod "d20477dc-2e11-4469-879d-eb1629c24898" (UID: "d20477dc-2e11-4469-879d-eb1629c24898"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:32:33.224574 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:33.224548 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d20477dc-2e11-4469-879d-eb1629c24898-kube-api-access-n9zfv" pod "d20477dc-2e11-4469-879d-eb1629c24898" (UID: "d20477dc-2e11-4469-879d-eb1629c24898"). InnerVolumeSpecName "kube-api-access-n9zfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:32:33.323545 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:33.323505 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9zfv\" (UniqueName: \"kubernetes.io/projected/d20477dc-2e11-4469-879d-eb1629c24898-kube-api-access-n9zfv\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:32:33.323545 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:33.323536 5935 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d20477dc-2e11-4469-879d-eb1629c24898-plugins-config-volume\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:32:33.858215 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:33.858175 5935 generic.go:356] "Generic (PLEG): container finished" podID="d20477dc-2e11-4469-879d-eb1629c24898" containerID="8e27bcd5732252c3ce733f5c6b01c9aa5989465f0c7eb4b8ef8ff77685711bcc" exitCode=0 Nov 01 20:32:33.858215 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:33.858218 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-a6873b-79b7f97879-9dhts" Nov 01 20:32:33.858780 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:33.858253 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-a6873b-79b7f97879-9dhts" event={"ID":"d20477dc-2e11-4469-879d-eb1629c24898","Type":"ContainerDied","Data":"8e27bcd5732252c3ce733f5c6b01c9aa5989465f0c7eb4b8ef8ff77685711bcc"} Nov 01 20:32:33.858780 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:33.858315 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-a6873b-79b7f97879-9dhts" event={"ID":"d20477dc-2e11-4469-879d-eb1629c24898","Type":"ContainerDied","Data":"e7d95fbf2ac02e096706c13fa0511752e086e8f750e100bd927d2aa236167cae"} Nov 01 20:32:33.858780 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:33.858337 5935 scope.go:122] "RemoveContainer" containerID="8e27bcd5732252c3ce733f5c6b01c9aa5989465f0c7eb4b8ef8ff77685711bcc" Nov 01 20:32:33.868531 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:33.868508 5935 scope.go:122] "RemoveContainer" containerID="8e27bcd5732252c3ce733f5c6b01c9aa5989465f0c7eb4b8ef8ff77685711bcc" Nov 01 20:32:33.868838 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:32:33.868819 5935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e27bcd5732252c3ce733f5c6b01c9aa5989465f0c7eb4b8ef8ff77685711bcc\": container with ID starting with 8e27bcd5732252c3ce733f5c6b01c9aa5989465f0c7eb4b8ef8ff77685711bcc not found: ID does not exist" containerID="8e27bcd5732252c3ce733f5c6b01c9aa5989465f0c7eb4b8ef8ff77685711bcc" Nov 01 20:32:33.868891 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:33.868850 5935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e27bcd5732252c3ce733f5c6b01c9aa5989465f0c7eb4b8ef8ff77685711bcc"} err="failed to get container status \"8e27bcd5732252c3ce733f5c6b01c9aa5989465f0c7eb4b8ef8ff77685711bcc\": rpc error: code = NotFound desc = could not find container \"8e27bcd5732252c3ce733f5c6b01c9aa5989465f0c7eb4b8ef8ff77685711bcc\": container with ID starting with 8e27bcd5732252c3ce733f5c6b01c9aa5989465f0c7eb4b8ef8ff77685711bcc not found: ID does not exist" Nov 01 20:32:33.882057 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:33.882021 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-a6873b-79b7f97879-9dhts"] Nov 01 20:32:33.885364 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:33.885339 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-a6873b-79b7f97879-9dhts"] Nov 01 20:32:33.926239 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:33.926153 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-29b57c-7b87bbfff-2nmvw" podUID="6465f212-e931-4b70-8b1a-a1ec15850515" containerName="payload-pre-processing" containerID="cri-o://01559ca3befe74277b2e30d17c2c8ecde908e6f86904ad7e3e0102bfa75abd94" gracePeriod=25 Nov 01 20:32:34.355073 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:34.355042 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42fe7bf5-319c-449a-aa5a-28b1e2d1f008" path="/var/lib/kubelet/pods/42fe7bf5-319c-449a-aa5a-28b1e2d1f008/volumes" Nov 01 20:32:34.355578 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:34.355558 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d20477dc-2e11-4469-879d-eb1629c24898" path="/var/lib/kubelet/pods/d20477dc-2e11-4469-879d-eb1629c24898/volumes" Nov 01 20:32:35.704843 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:35.704803 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn"] Nov 01 20:32:35.705301 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:35.705277 5935 cpu_manager.go:414] "containerMap: removing container" podUID="c5b342ce-508a-445c-8034-ff83d26a7190" containerName="cleanup" Nov 01 20:32:35.705345 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:35.705302 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5b342ce-508a-445c-8034-ff83d26a7190" containerName="cleanup" Nov 01 20:32:35.705345 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:35.705320 5935 cpu_manager.go:414] "containerMap: removing container" podUID="52680409-4495-4000-ade4-4498da85c23b" containerName="discovery" Nov 01 20:32:35.705345 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:35.705325 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52680409-4495-4000-ade4-4498da85c23b" containerName="discovery" Nov 01 20:32:35.705345 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:35.705336 5935 cpu_manager.go:414] "containerMap: removing container" podUID="c5b342ce-508a-445c-8034-ff83d26a7190" containerName="cleanup" Nov 01 20:32:35.705345 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:35.705342 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5b342ce-508a-445c-8034-ff83d26a7190" containerName="cleanup" Nov 01 20:32:35.705498 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:35.705349 5935 cpu_manager.go:414] "containerMap: removing container" podUID="7cdc5a94-7c68-49f2-95b9-8d23d1355581" containerName="maas-api" Nov 01 20:32:35.705498 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:35.705383 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cdc5a94-7c68-49f2-95b9-8d23d1355581" containerName="maas-api" Nov 01 20:32:35.705498 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:35.705395 5935 cpu_manager.go:414] "containerMap: removing container" podUID="92a72792-1735-4970-88ac-ca2a50a5b63e" containerName="istio-proxy" Nov 01 20:32:35.705498 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:35.705400 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92a72792-1735-4970-88ac-ca2a50a5b63e" containerName="istio-proxy" Nov 01 20:32:35.705498 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:35.705407 5935 cpu_manager.go:414] "containerMap: removing container" podUID="e2b197fc-6182-4f88-8e11-d631a9d95b37" containerName="istio-proxy" Nov 01 20:32:35.705498 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:35.705412 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2b197fc-6182-4f88-8e11-d631a9d95b37" containerName="istio-proxy" Nov 01 20:32:35.705498 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:35.705417 5935 cpu_manager.go:414] "containerMap: removing container" podUID="5cb9e245-7141-402d-9601-6f374e26fff3" containerName="istio-proxy" Nov 01 20:32:35.705498 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:35.705435 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cb9e245-7141-402d-9601-6f374e26fff3" containerName="istio-proxy" Nov 01 20:32:35.705498 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:35.705451 5935 cpu_manager.go:414] "containerMap: removing container" podUID="50eb270e-173d-4727-9504-1b69379f7f77" containerName="main" Nov 01 20:32:35.705498 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:35.705457 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50eb270e-173d-4727-9504-1b69379f7f77" containerName="main" Nov 01 20:32:35.705498 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:35.705485 5935 cpu_manager.go:414] "containerMap: removing container" podUID="9e1d573b-e388-4d7b-b47a-8a01467c03b2" containerName="main" Nov 01 20:32:35.705498 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:35.705492 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e1d573b-e388-4d7b-b47a-8a01467c03b2" containerName="main" Nov 01 20:32:35.705903 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:35.705505 5935 cpu_manager.go:414] "containerMap: removing container" podUID="56d6fb4f-c372-490f-9dec-5b36a2ab21e5" containerName="maas-api" Nov 01 20:32:35.705903 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:35.705512 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56d6fb4f-c372-490f-9dec-5b36a2ab21e5" containerName="maas-api" Nov 01 20:32:35.705903 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:35.705521 5935 cpu_manager.go:414] "containerMap: removing container" podUID="d20477dc-2e11-4469-879d-eb1629c24898" containerName="payload-processing" Nov 01 20:32:35.705903 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:35.705528 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d20477dc-2e11-4469-879d-eb1629c24898" containerName="payload-processing" Nov 01 20:32:35.705903 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:35.705543 5935 cpu_manager.go:414] "containerMap: removing container" podUID="42fe7bf5-319c-449a-aa5a-28b1e2d1f008" containerName="maas-api" Nov 01 20:32:35.705903 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:35.705550 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42fe7bf5-319c-449a-aa5a-28b1e2d1f008" containerName="maas-api" Nov 01 20:32:35.705903 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:35.705558 5935 cpu_manager.go:414] "containerMap: removing container" podUID="ab7a8222-c059-4ecd-8187-fb711ac219c4" containerName="istio-proxy" Nov 01 20:32:35.705903 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:35.705565 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab7a8222-c059-4ecd-8187-fb711ac219c4" containerName="istio-proxy" Nov 01 20:32:35.705903 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:35.705686 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="50eb270e-173d-4727-9504-1b69379f7f77" containerName="main" Nov 01 20:32:35.705903 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:35.705701 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5b342ce-508a-445c-8034-ff83d26a7190" containerName="cleanup" Nov 01 20:32:35.705903 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:35.705712 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e1d573b-e388-4d7b-b47a-8a01467c03b2" containerName="main" Nov 01 20:32:35.705903 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:35.705720 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="56d6fb4f-c372-490f-9dec-5b36a2ab21e5" containerName="maas-api" Nov 01 20:32:35.705903 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:35.705730 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="d20477dc-2e11-4469-879d-eb1629c24898" containerName="payload-processing" Nov 01 20:32:35.705903 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:35.705738 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="42fe7bf5-319c-449a-aa5a-28b1e2d1f008" containerName="maas-api" Nov 01 20:32:35.705903 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:35.705748 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab7a8222-c059-4ecd-8187-fb711ac219c4" containerName="istio-proxy" Nov 01 20:32:35.705903 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:35.705756 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5b342ce-508a-445c-8034-ff83d26a7190" containerName="cleanup" Nov 01 20:32:35.705903 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:35.705766 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="52680409-4495-4000-ade4-4498da85c23b" containerName="discovery" Nov 01 20:32:35.705903 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:35.705776 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5b342ce-508a-445c-8034-ff83d26a7190" containerName="cleanup" Nov 01 20:32:35.705903 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:35.705784 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cdc5a94-7c68-49f2-95b9-8d23d1355581" containerName="maas-api" Nov 01 20:32:35.705903 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:35.705792 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="92a72792-1735-4970-88ac-ca2a50a5b63e" containerName="istio-proxy" Nov 01 20:32:35.705903 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:35.705801 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2b197fc-6182-4f88-8e11-d631a9d95b37" containerName="istio-proxy" Nov 01 20:32:35.705903 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:35.705809 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cb9e245-7141-402d-9601-6f374e26fff3" containerName="istio-proxy" Nov 01 20:32:35.710674 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:35.710651 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn" Nov 01 20:32:35.718973 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:35.718948 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-a6873b-openshift-default-dockercfg-hfgdz\"" Nov 01 20:32:35.722768 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:35.722744 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn"] Nov 01 20:32:35.843819 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:32:35.843787 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:32:35.853398 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:35.853371 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20125155-e034-40cd-b7dc-3bc9aab77ab8-istiod-ca-cert\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn\" (UID: \"20125155-e034-40cd-b7dc-3bc9aab77ab8\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn" Nov 01 20:32:35.853495 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:35.853428 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20125155-e034-40cd-b7dc-3bc9aab77ab8-credential-socket\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn\" (UID: \"20125155-e034-40cd-b7dc-3bc9aab77ab8\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn" Nov 01 20:32:35.853560 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:35.853547 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20125155-e034-40cd-b7dc-3bc9aab77ab8-istio-envoy\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn\" (UID: \"20125155-e034-40cd-b7dc-3bc9aab77ab8\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn" Nov 01 20:32:35.853609 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:35.853595 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20125155-e034-40cd-b7dc-3bc9aab77ab8-workload-socket\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn\" (UID: \"20125155-e034-40cd-b7dc-3bc9aab77ab8\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn" Nov 01 20:32:35.853649 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:35.853615 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20125155-e034-40cd-b7dc-3bc9aab77ab8-workload-certs\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn\" (UID: \"20125155-e034-40cd-b7dc-3bc9aab77ab8\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn" Nov 01 20:32:35.853649 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:35.853636 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20125155-e034-40cd-b7dc-3bc9aab77ab8-istio-data\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn\" (UID: \"20125155-e034-40cd-b7dc-3bc9aab77ab8\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn" Nov 01 20:32:35.853725 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:35.853698 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20125155-e034-40cd-b7dc-3bc9aab77ab8-istio-podinfo\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn\" (UID: \"20125155-e034-40cd-b7dc-3bc9aab77ab8\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn" Nov 01 20:32:35.853785 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:35.853771 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20125155-e034-40cd-b7dc-3bc9aab77ab8-istio-token\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn\" (UID: \"20125155-e034-40cd-b7dc-3bc9aab77ab8\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn" Nov 01 20:32:35.853829 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:35.853809 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-429sb\" (UniqueName: \"kubernetes.io/projected/20125155-e034-40cd-b7dc-3bc9aab77ab8-kube-api-access-429sb\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn\" (UID: \"20125155-e034-40cd-b7dc-3bc9aab77ab8\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn" Nov 01 20:32:35.955385 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:35.955271 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20125155-e034-40cd-b7dc-3bc9aab77ab8-workload-socket\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn\" (UID: \"20125155-e034-40cd-b7dc-3bc9aab77ab8\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn" Nov 01 20:32:35.955385 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:35.955343 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20125155-e034-40cd-b7dc-3bc9aab77ab8-workload-certs\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn\" (UID: \"20125155-e034-40cd-b7dc-3bc9aab77ab8\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn" Nov 01 20:32:35.955385 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:35.955379 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20125155-e034-40cd-b7dc-3bc9aab77ab8-istio-data\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn\" (UID: \"20125155-e034-40cd-b7dc-3bc9aab77ab8\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn" Nov 01 20:32:35.955691 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:35.955401 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20125155-e034-40cd-b7dc-3bc9aab77ab8-istio-podinfo\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn\" (UID: \"20125155-e034-40cd-b7dc-3bc9aab77ab8\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn" Nov 01 20:32:35.955691 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:35.955452 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20125155-e034-40cd-b7dc-3bc9aab77ab8-istio-token\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn\" (UID: \"20125155-e034-40cd-b7dc-3bc9aab77ab8\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn" Nov 01 20:32:35.955691 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:35.955488 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-429sb\" (UniqueName: \"kubernetes.io/projected/20125155-e034-40cd-b7dc-3bc9aab77ab8-kube-api-access-429sb\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn\" (UID: \"20125155-e034-40cd-b7dc-3bc9aab77ab8\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn" Nov 01 20:32:35.955841 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:35.955788 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20125155-e034-40cd-b7dc-3bc9aab77ab8-workload-socket\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn\" (UID: \"20125155-e034-40cd-b7dc-3bc9aab77ab8\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn" Nov 01 20:32:35.955918 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:35.955899 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20125155-e034-40cd-b7dc-3bc9aab77ab8-istiod-ca-cert\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn\" (UID: \"20125155-e034-40cd-b7dc-3bc9aab77ab8\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn" Nov 01 20:32:35.955970 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:35.955947 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20125155-e034-40cd-b7dc-3bc9aab77ab8-workload-certs\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn\" (UID: \"20125155-e034-40cd-b7dc-3bc9aab77ab8\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn" Nov 01 20:32:35.956004 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:35.955976 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20125155-e034-40cd-b7dc-3bc9aab77ab8-istio-data\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn\" (UID: \"20125155-e034-40cd-b7dc-3bc9aab77ab8\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn" Nov 01 20:32:35.956004 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:35.955995 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20125155-e034-40cd-b7dc-3bc9aab77ab8-credential-socket\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn\" (UID: \"20125155-e034-40cd-b7dc-3bc9aab77ab8\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn" Nov 01 20:32:35.956076 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:35.956064 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20125155-e034-40cd-b7dc-3bc9aab77ab8-istio-envoy\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn\" (UID: \"20125155-e034-40cd-b7dc-3bc9aab77ab8\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn" Nov 01 20:32:35.956272 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:35.956246 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20125155-e034-40cd-b7dc-3bc9aab77ab8-credential-socket\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn\" (UID: \"20125155-e034-40cd-b7dc-3bc9aab77ab8\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn" Nov 01 20:32:35.956383 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:35.956321 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20125155-e034-40cd-b7dc-3bc9aab77ab8-istiod-ca-cert\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn\" (UID: \"20125155-e034-40cd-b7dc-3bc9aab77ab8\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn" Nov 01 20:32:35.958117 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:35.958095 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20125155-e034-40cd-b7dc-3bc9aab77ab8-istio-podinfo\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn\" (UID: \"20125155-e034-40cd-b7dc-3bc9aab77ab8\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn" Nov 01 20:32:35.958198 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:35.958112 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20125155-e034-40cd-b7dc-3bc9aab77ab8-istio-envoy\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn\" (UID: \"20125155-e034-40cd-b7dc-3bc9aab77ab8\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn" Nov 01 20:32:35.964243 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:35.964046 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20125155-e034-40cd-b7dc-3bc9aab77ab8-istio-token\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn\" (UID: \"20125155-e034-40cd-b7dc-3bc9aab77ab8\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn" Nov 01 20:32:35.964578 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:35.964541 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-429sb\" (UniqueName: \"kubernetes.io/projected/20125155-e034-40cd-b7dc-3bc9aab77ab8-kube-api-access-429sb\") pod \"e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn\" (UID: \"20125155-e034-40cd-b7dc-3bc9aab77ab8\") " pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn" Nov 01 20:32:36.026297 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:36.026247 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn" Nov 01 20:32:36.308128 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:36.308101 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn"] Nov 01 20:32:36.310228 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:32:36.310200 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20125155_e034_40cd_b7dc_3bc9aab77ab8.slice/crio-4be3b18261fd7e81ccf7d5a8ec385246a75a9624a99e98716cece1934ed31e7f WatchSource:0}: Error finding container 4be3b18261fd7e81ccf7d5a8ec385246a75a9624a99e98716cece1934ed31e7f: Status 404 returned error can't find the container with id 4be3b18261fd7e81ccf7d5a8ec385246a75a9624a99e98716cece1934ed31e7f Nov 01 20:32:36.312684 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:36.312650 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:32:36.312773 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:36.312720 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:32:36.312773 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:36.312761 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:32:36.571632 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:36.571550 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-29b57c-7b87bbfff-2nmvw" podUID="6465f212-e931-4b70-8b1a-a1ec15850515" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 20:32:36.882597 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:36.882499 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn" event={"ID":"20125155-e034-40cd-b7dc-3bc9aab77ab8","Type":"ContainerStarted","Data":"6d4b0a5e42bf3062afae776def7ab2b0b19a386b44920be78277a4dacedd4666"} Nov 01 20:32:36.882597 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:36.882560 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn" event={"ID":"20125155-e034-40cd-b7dc-3bc9aab77ab8","Type":"ContainerStarted","Data":"4be3b18261fd7e81ccf7d5a8ec385246a75a9624a99e98716cece1934ed31e7f"} Nov 01 20:32:36.901568 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:36.901511 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn" podStartSLOduration=1.901495035 podStartE2EDuration="1.901495035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:32:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:32:36.900763838 +0000 UTC m=+2002.673924361" watchObservedRunningTime="2026-11-01 20:32:36.901495035 +0000 UTC m=+2002.674655547" Nov 01 20:32:37.027004 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:37.026961 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn" Nov 01 20:32:37.028583 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:37.028556 5935 patch_prober.go:29] interesting pod/e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 01 20:32:37.028743 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:37.028630 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn" podUID="20125155-e034-40cd-b7dc-3bc9aab77ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 01 20:32:38.027569 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:38.027535 5935 patch_prober.go:29] interesting pod/e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 01 20:32:38.028054 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:38.027595 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn" podUID="20125155-e034-40cd-b7dc-3bc9aab77ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 01 20:32:39.027613 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:39.027571 5935 patch_prober.go:29] interesting pod/e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 01 20:32:39.028061 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:39.027630 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn" podUID="20125155-e034-40cd-b7dc-3bc9aab77ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 01 20:32:40.027097 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:40.027062 5935 patch_prober.go:29] interesting pod/e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 01 20:32:40.027334 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:40.027127 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn" podUID="20125155-e034-40cd-b7dc-3bc9aab77ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 01 20:32:40.738503 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:40.738475 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-036c69-79cd8c8b6f-fxpb5" Nov 01 20:32:40.834876 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:40.834839 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/624bb371-6040-4cd4-a668-a8afc016d132-kube-api-access-f5hnt\" (UniqueName: \"kubernetes.io/projected/624bb371-6040-4cd4-a668-a8afc016d132-kube-api-access-f5hnt\") pod \"624bb371-6040-4cd4-a668-a8afc016d132\" (UID: \"624bb371-6040-4cd4-a668-a8afc016d132\") " Nov 01 20:32:40.835054 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:40.834938 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/624bb371-6040-4cd4-a668-a8afc016d132-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/624bb371-6040-4cd4-a668-a8afc016d132-plugins-config-volume\") pod \"624bb371-6040-4cd4-a668-a8afc016d132\" (UID: \"624bb371-6040-4cd4-a668-a8afc016d132\") " Nov 01 20:32:40.835405 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:40.835384 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/624bb371-6040-4cd4-a668-a8afc016d132-plugins-config-volume" pod "624bb371-6040-4cd4-a668-a8afc016d132" (UID: "624bb371-6040-4cd4-a668-a8afc016d132"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:32:40.837043 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:40.837027 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/624bb371-6040-4cd4-a668-a8afc016d132-kube-api-access-f5hnt" pod "624bb371-6040-4cd4-a668-a8afc016d132" (UID: "624bb371-6040-4cd4-a668-a8afc016d132"). InnerVolumeSpecName "kube-api-access-f5hnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:32:40.918688 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:40.918648 5935 generic.go:356] "Generic (PLEG): container finished" podID="624bb371-6040-4cd4-a668-a8afc016d132" containerID="fe2827cfb5d9c4adf9c8cb52d85dcfafbd21e70d88b11f1fa46cd667fe43c447" exitCode=0 Nov 01 20:32:40.918846 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:40.918701 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-036c69-79cd8c8b6f-fxpb5" Nov 01 20:32:40.918846 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:40.918730 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-036c69-79cd8c8b6f-fxpb5" event={"ID":"624bb371-6040-4cd4-a668-a8afc016d132","Type":"ContainerDied","Data":"fe2827cfb5d9c4adf9c8cb52d85dcfafbd21e70d88b11f1fa46cd667fe43c447"} Nov 01 20:32:40.918846 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:40.918790 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-036c69-79cd8c8b6f-fxpb5" event={"ID":"624bb371-6040-4cd4-a668-a8afc016d132","Type":"ContainerDied","Data":"18c917bb9be838675304ecb6a613727e657cdcc08c3b0068819c43c2428174ad"} Nov 01 20:32:40.918846 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:40.918813 5935 scope.go:122] "RemoveContainer" containerID="fe2827cfb5d9c4adf9c8cb52d85dcfafbd21e70d88b11f1fa46cd667fe43c447" Nov 01 20:32:40.928715 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:40.928699 5935 scope.go:122] "RemoveContainer" containerID="fe2827cfb5d9c4adf9c8cb52d85dcfafbd21e70d88b11f1fa46cd667fe43c447" Nov 01 20:32:40.929009 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:32:40.928990 5935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe2827cfb5d9c4adf9c8cb52d85dcfafbd21e70d88b11f1fa46cd667fe43c447\": container with ID starting with fe2827cfb5d9c4adf9c8cb52d85dcfafbd21e70d88b11f1fa46cd667fe43c447 not found: ID does not exist" containerID="fe2827cfb5d9c4adf9c8cb52d85dcfafbd21e70d88b11f1fa46cd667fe43c447" Nov 01 20:32:40.929058 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:40.929019 5935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe2827cfb5d9c4adf9c8cb52d85dcfafbd21e70d88b11f1fa46cd667fe43c447"} err="failed to get container status \"fe2827cfb5d9c4adf9c8cb52d85dcfafbd21e70d88b11f1fa46cd667fe43c447\": rpc error: code = NotFound desc = could not find container \"fe2827cfb5d9c4adf9c8cb52d85dcfafbd21e70d88b11f1fa46cd667fe43c447\": container with ID starting with fe2827cfb5d9c4adf9c8cb52d85dcfafbd21e70d88b11f1fa46cd667fe43c447 not found: ID does not exist" Nov 01 20:32:40.936143 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:40.936120 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5hnt\" (UniqueName: \"kubernetes.io/projected/624bb371-6040-4cd4-a668-a8afc016d132-kube-api-access-f5hnt\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:32:40.936143 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:40.936142 5935 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/624bb371-6040-4cd4-a668-a8afc016d132-plugins-config-volume\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:32:40.940332 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:40.940308 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-036c69-79cd8c8b6f-fxpb5"] Nov 01 20:32:40.958311 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:40.945176 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-036c69-79cd8c8b6f-fxpb5"] Nov 01 20:32:41.027624 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:41.027542 5935 patch_prober.go:29] interesting pod/e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 01 20:32:41.027624 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:41.027609 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn" podUID="20125155-e034-40cd-b7dc-3bc9aab77ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 01 20:32:42.026827 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:42.026764 5935 patch_prober.go:29] interesting pod/e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 01 20:32:42.026827 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:42.026824 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn" podUID="20125155-e034-40cd-b7dc-3bc9aab77ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 01 20:32:42.378889 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:42.378796 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="624bb371-6040-4cd4-a668-a8afc016d132" path="/var/lib/kubelet/pods/624bb371-6040-4cd4-a668-a8afc016d132/volumes" Nov 01 20:32:43.027246 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:43.027211 5935 patch_prober.go:29] interesting pod/e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Nov 01 20:32:43.027809 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:43.027277 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn" podUID="20125155-e034-40cd-b7dc-3bc9aab77ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Nov 01 20:32:44.031794 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:44.031760 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn" Nov 01 20:32:44.032317 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:44.032207 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn" Nov 01 20:32:44.032893 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:44.032870 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn" Nov 01 20:32:45.135772 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:45.135738 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-bccce1-5b9f8747f9-rvbs2"] Nov 01 20:32:46.572427 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:46.572393 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-29b57c-7b87bbfff-2nmvw" podUID="6465f212-e931-4b70-8b1a-a1ec15850515" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 20:32:50.137071 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:50.137018 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-bccce1-5b9f8747f9-rvbs2" podUID="de43c730-bc16-43b6-8cbb-c32a14622bb7" containerName="payload-processing" containerID="cri-o://41a73ca59d45b4e8ff308e407275a2ca069412843d1c74cb0b587e30976c0947" gracePeriod=25 Nov 01 20:32:50.399550 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:50.399481 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-bccce1-5b9f8747f9-rvbs2" Nov 01 20:32:50.468486 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:50.468448 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de43c730-bc16-43b6-8cbb-c32a14622bb7-kube-api-access-trmkv\" (UniqueName: \"kubernetes.io/projected/de43c730-bc16-43b6-8cbb-c32a14622bb7-kube-api-access-trmkv\") pod \"de43c730-bc16-43b6-8cbb-c32a14622bb7\" (UID: \"de43c730-bc16-43b6-8cbb-c32a14622bb7\") " Nov 01 20:32:50.468644 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:50.468557 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de43c730-bc16-43b6-8cbb-c32a14622bb7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de43c730-bc16-43b6-8cbb-c32a14622bb7-plugins-config-volume\") pod \"de43c730-bc16-43b6-8cbb-c32a14622bb7\" (UID: \"de43c730-bc16-43b6-8cbb-c32a14622bb7\") " Nov 01 20:32:50.469021 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:50.468987 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de43c730-bc16-43b6-8cbb-c32a14622bb7-plugins-config-volume" pod "de43c730-bc16-43b6-8cbb-c32a14622bb7" (UID: "de43c730-bc16-43b6-8cbb-c32a14622bb7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:32:50.469182 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:50.469119 5935 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de43c730-bc16-43b6-8cbb-c32a14622bb7-plugins-config-volume\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:32:50.470702 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:50.470676 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de43c730-bc16-43b6-8cbb-c32a14622bb7-kube-api-access-trmkv" pod "de43c730-bc16-43b6-8cbb-c32a14622bb7" (UID: "de43c730-bc16-43b6-8cbb-c32a14622bb7"). InnerVolumeSpecName "kube-api-access-trmkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:32:50.570107 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:50.570058 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trmkv\" (UniqueName: \"kubernetes.io/projected/de43c730-bc16-43b6-8cbb-c32a14622bb7-kube-api-access-trmkv\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:32:50.981953 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:50.981919 5935 generic.go:356] "Generic (PLEG): container finished" podID="de43c730-bc16-43b6-8cbb-c32a14622bb7" containerID="41a73ca59d45b4e8ff308e407275a2ca069412843d1c74cb0b587e30976c0947" exitCode=0 Nov 01 20:32:50.982167 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:50.981969 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-bccce1-5b9f8747f9-rvbs2" Nov 01 20:32:50.982167 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:50.982033 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-bccce1-5b9f8747f9-rvbs2" event={"ID":"de43c730-bc16-43b6-8cbb-c32a14622bb7","Type":"ContainerDied","Data":"41a73ca59d45b4e8ff308e407275a2ca069412843d1c74cb0b587e30976c0947"} Nov 01 20:32:50.982167 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:50.982097 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-bccce1-5b9f8747f9-rvbs2" event={"ID":"de43c730-bc16-43b6-8cbb-c32a14622bb7","Type":"ContainerDied","Data":"73faa839da9433ee6e251be9cf345bf90534fab37e76cf24eccbbb3c6c369d49"} Nov 01 20:32:50.982167 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:50.982130 5935 scope.go:122] "RemoveContainer" containerID="41a73ca59d45b4e8ff308e407275a2ca069412843d1c74cb0b587e30976c0947" Nov 01 20:32:50.996802 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:50.996780 5935 scope.go:122] "RemoveContainer" containerID="41a73ca59d45b4e8ff308e407275a2ca069412843d1c74cb0b587e30976c0947" Nov 01 20:32:50.997197 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:32:50.997160 5935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41a73ca59d45b4e8ff308e407275a2ca069412843d1c74cb0b587e30976c0947\": container with ID starting with 41a73ca59d45b4e8ff308e407275a2ca069412843d1c74cb0b587e30976c0947 not found: ID does not exist" containerID="41a73ca59d45b4e8ff308e407275a2ca069412843d1c74cb0b587e30976c0947" Nov 01 20:32:50.997328 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:50.997207 5935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41a73ca59d45b4e8ff308e407275a2ca069412843d1c74cb0b587e30976c0947"} err="failed to get container status \"41a73ca59d45b4e8ff308e407275a2ca069412843d1c74cb0b587e30976c0947\": rpc error: code = NotFound desc = could not find container \"41a73ca59d45b4e8ff308e407275a2ca069412843d1c74cb0b587e30976c0947\": container with ID starting with 41a73ca59d45b4e8ff308e407275a2ca069412843d1c74cb0b587e30976c0947 not found: ID does not exist" Nov 01 20:32:51.005411 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:51.005381 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-bccce1-5b9f8747f9-rvbs2"] Nov 01 20:32:51.007463 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:51.007435 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-bccce1-5b9f8747f9-rvbs2"] Nov 01 20:32:52.353299 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:52.353257 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de43c730-bc16-43b6-8cbb-c32a14622bb7" path="/var/lib/kubelet/pods/de43c730-bc16-43b6-8cbb-c32a14622bb7/volumes" Nov 01 20:32:54.237787 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:54.237757 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-29b57c-7b87bbfff-2nmvw" Nov 01 20:32:54.324609 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:54.323744 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6465f212-e931-4b70-8b1a-a1ec15850515-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6465f212-e931-4b70-8b1a-a1ec15850515-plugins-config-volume\") pod \"6465f212-e931-4b70-8b1a-a1ec15850515\" (UID: \"6465f212-e931-4b70-8b1a-a1ec15850515\") " Nov 01 20:32:54.324609 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:54.323822 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6465f212-e931-4b70-8b1a-a1ec15850515-kube-api-access-w9k2l\" (UniqueName: \"kubernetes.io/projected/6465f212-e931-4b70-8b1a-a1ec15850515-kube-api-access-w9k2l\") pod \"6465f212-e931-4b70-8b1a-a1ec15850515\" (UID: \"6465f212-e931-4b70-8b1a-a1ec15850515\") " Nov 01 20:32:54.324609 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:54.324189 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6465f212-e931-4b70-8b1a-a1ec15850515-plugins-config-volume" pod "6465f212-e931-4b70-8b1a-a1ec15850515" (UID: "6465f212-e931-4b70-8b1a-a1ec15850515"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:32:54.326997 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:54.326964 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6465f212-e931-4b70-8b1a-a1ec15850515-kube-api-access-w9k2l" pod "6465f212-e931-4b70-8b1a-a1ec15850515" (UID: "6465f212-e931-4b70-8b1a-a1ec15850515"). InnerVolumeSpecName "kube-api-access-w9k2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:32:54.424889 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:54.424851 5935 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6465f212-e931-4b70-8b1a-a1ec15850515-plugins-config-volume\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:32:54.424889 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:54.424884 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9k2l\" (UniqueName: \"kubernetes.io/projected/6465f212-e931-4b70-8b1a-a1ec15850515-kube-api-access-w9k2l\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:32:55.009998 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:55.009962 5935 generic.go:356] "Generic (PLEG): container finished" podID="6465f212-e931-4b70-8b1a-a1ec15850515" containerID="01559ca3befe74277b2e30d17c2c8ecde908e6f86904ad7e3e0102bfa75abd94" exitCode=0 Nov 01 20:32:55.010212 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:55.010190 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-29b57c-7b87bbfff-2nmvw" event={"ID":"6465f212-e931-4b70-8b1a-a1ec15850515","Type":"ContainerDied","Data":"01559ca3befe74277b2e30d17c2c8ecde908e6f86904ad7e3e0102bfa75abd94"} Nov 01 20:32:55.010270 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:55.010248 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-29b57c-7b87bbfff-2nmvw" event={"ID":"6465f212-e931-4b70-8b1a-a1ec15850515","Type":"ContainerDied","Data":"acfbb3063db32daddb0e0024d27d3b0bf0b2437573349b8c0ce1683922194841"} Nov 01 20:32:55.010384 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:55.010280 5935 scope.go:122] "RemoveContainer" containerID="01559ca3befe74277b2e30d17c2c8ecde908e6f86904ad7e3e0102bfa75abd94" Nov 01 20:32:55.010483 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:55.010469 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-29b57c-7b87bbfff-2nmvw" Nov 01 20:32:55.023599 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:55.023575 5935 scope.go:122] "RemoveContainer" containerID="01559ca3befe74277b2e30d17c2c8ecde908e6f86904ad7e3e0102bfa75abd94" Nov 01 20:32:55.023958 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:32:55.023933 5935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01559ca3befe74277b2e30d17c2c8ecde908e6f86904ad7e3e0102bfa75abd94\": container with ID starting with 01559ca3befe74277b2e30d17c2c8ecde908e6f86904ad7e3e0102bfa75abd94 not found: ID does not exist" containerID="01559ca3befe74277b2e30d17c2c8ecde908e6f86904ad7e3e0102bfa75abd94" Nov 01 20:32:55.024025 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:55.023968 5935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01559ca3befe74277b2e30d17c2c8ecde908e6f86904ad7e3e0102bfa75abd94"} err="failed to get container status \"01559ca3befe74277b2e30d17c2c8ecde908e6f86904ad7e3e0102bfa75abd94\": rpc error: code = NotFound desc = could not find container \"01559ca3befe74277b2e30d17c2c8ecde908e6f86904ad7e3e0102bfa75abd94\": container with ID starting with 01559ca3befe74277b2e30d17c2c8ecde908e6f86904ad7e3e0102bfa75abd94 not found: ID does not exist" Nov 01 20:32:55.028168 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:55.028142 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-29b57c-7b87bbfff-2nmvw"] Nov 01 20:32:55.034325 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:55.034303 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-29b57c-7b87bbfff-2nmvw"] Nov 01 20:32:56.353183 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:56.353149 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6465f212-e931-4b70-8b1a-a1ec15850515" path="/var/lib/kubelet/pods/6465f212-e931-4b70-8b1a-a1ec15850515/volumes" Nov 01 20:32:57.069588 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:57.069556 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-0aa61b/distinct-w4-0aa61b-kserve-6674df959d-2lm95"] Nov 01 20:32:57.090419 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:32:57.090385 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-0aa61b/simulator-w4-0aa61b-kserve-6774594d4b-qmjz8"] Nov 01 20:33:01.712518 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:01.712474 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn"] Nov 01 20:33:01.713064 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:01.712727 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn" podUID="20125155-e034-40cd-b7dc-3bc9aab77ab8" containerName="istio-proxy" containerID="cri-o://6d4b0a5e42bf3062afae776def7ab2b0b19a386b44920be78277a4dacedd4666" gracePeriod=30 Nov 01 20:33:03.298886 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:03.298839 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-0aa61b-5c479c87bc-jsktt"] Nov 01 20:33:05.844486 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:33:05.844438 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:33:06.984187 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:06.984126 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn" Nov 01 20:33:07.065856 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:07.065813 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20125155-e034-40cd-b7dc-3bc9aab77ab8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20125155-e034-40cd-b7dc-3bc9aab77ab8-istiod-ca-cert\") pod \"20125155-e034-40cd-b7dc-3bc9aab77ab8\" (UID: \"20125155-e034-40cd-b7dc-3bc9aab77ab8\") " Nov 01 20:33:07.066057 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:07.065917 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20125155-e034-40cd-b7dc-3bc9aab77ab8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20125155-e034-40cd-b7dc-3bc9aab77ab8-credential-socket\") pod \"20125155-e034-40cd-b7dc-3bc9aab77ab8\" (UID: \"20125155-e034-40cd-b7dc-3bc9aab77ab8\") " Nov 01 20:33:07.066135 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:07.066079 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20125155-e034-40cd-b7dc-3bc9aab77ab8-kube-api-access-429sb\" (UniqueName: \"kubernetes.io/projected/20125155-e034-40cd-b7dc-3bc9aab77ab8-kube-api-access-429sb\") pod \"20125155-e034-40cd-b7dc-3bc9aab77ab8\" (UID: \"20125155-e034-40cd-b7dc-3bc9aab77ab8\") " Nov 01 20:33:07.066135 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:07.066114 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20125155-e034-40cd-b7dc-3bc9aab77ab8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20125155-e034-40cd-b7dc-3bc9aab77ab8-istio-envoy\") pod \"20125155-e034-40cd-b7dc-3bc9aab77ab8\" (UID: \"20125155-e034-40cd-b7dc-3bc9aab77ab8\") " Nov 01 20:33:07.066251 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:07.066196 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20125155-e034-40cd-b7dc-3bc9aab77ab8-credential-socket" pod "20125155-e034-40cd-b7dc-3bc9aab77ab8" (UID: "20125155-e034-40cd-b7dc-3bc9aab77ab8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:07.066251 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:07.066206 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20125155-e034-40cd-b7dc-3bc9aab77ab8-istio-token\" (UniqueName: \"kubernetes.io/projected/20125155-e034-40cd-b7dc-3bc9aab77ab8-istio-token\") pod \"20125155-e034-40cd-b7dc-3bc9aab77ab8\" (UID: \"20125155-e034-40cd-b7dc-3bc9aab77ab8\") " Nov 01 20:33:07.066251 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:07.066243 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20125155-e034-40cd-b7dc-3bc9aab77ab8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20125155-e034-40cd-b7dc-3bc9aab77ab8-workload-socket\") pod \"20125155-e034-40cd-b7dc-3bc9aab77ab8\" (UID: \"20125155-e034-40cd-b7dc-3bc9aab77ab8\") " Nov 01 20:33:07.066494 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:07.066270 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/20125155-e034-40cd-b7dc-3bc9aab77ab8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20125155-e034-40cd-b7dc-3bc9aab77ab8-istio-podinfo\") pod \"20125155-e034-40cd-b7dc-3bc9aab77ab8\" (UID: \"20125155-e034-40cd-b7dc-3bc9aab77ab8\") " Nov 01 20:33:07.066494 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:07.066337 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20125155-e034-40cd-b7dc-3bc9aab77ab8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20125155-e034-40cd-b7dc-3bc9aab77ab8-istio-data\") pod \"20125155-e034-40cd-b7dc-3bc9aab77ab8\" (UID: \"20125155-e034-40cd-b7dc-3bc9aab77ab8\") " Nov 01 20:33:07.066494 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:07.066340 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20125155-e034-40cd-b7dc-3bc9aab77ab8-istiod-ca-cert" pod "20125155-e034-40cd-b7dc-3bc9aab77ab8" (UID: "20125155-e034-40cd-b7dc-3bc9aab77ab8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:33:07.066494 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:07.066460 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20125155-e034-40cd-b7dc-3bc9aab77ab8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20125155-e034-40cd-b7dc-3bc9aab77ab8-workload-certs\") pod \"20125155-e034-40cd-b7dc-3bc9aab77ab8\" (UID: \"20125155-e034-40cd-b7dc-3bc9aab77ab8\") " Nov 01 20:33:07.066733 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:07.066710 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20125155-e034-40cd-b7dc-3bc9aab77ab8-istio-data" pod "20125155-e034-40cd-b7dc-3bc9aab77ab8" (UID: "20125155-e034-40cd-b7dc-3bc9aab77ab8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:07.066883 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:07.066862 5935 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20125155-e034-40cd-b7dc-3bc9aab77ab8-istiod-ca-cert\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:33:07.066952 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:07.066891 5935 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20125155-e034-40cd-b7dc-3bc9aab77ab8-credential-socket\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:33:07.066952 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:07.066892 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20125155-e034-40cd-b7dc-3bc9aab77ab8-workload-certs" pod "20125155-e034-40cd-b7dc-3bc9aab77ab8" (UID: "20125155-e034-40cd-b7dc-3bc9aab77ab8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:07.066952 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:07.066905 5935 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20125155-e034-40cd-b7dc-3bc9aab77ab8-istio-data\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:33:07.066952 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:07.066914 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20125155-e034-40cd-b7dc-3bc9aab77ab8-workload-socket" pod "20125155-e034-40cd-b7dc-3bc9aab77ab8" (UID: "20125155-e034-40cd-b7dc-3bc9aab77ab8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:07.068538 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:07.068508 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20125155-e034-40cd-b7dc-3bc9aab77ab8-istio-envoy" pod "20125155-e034-40cd-b7dc-3bc9aab77ab8" (UID: "20125155-e034-40cd-b7dc-3bc9aab77ab8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:07.068865 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:07.068846 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20125155-e034-40cd-b7dc-3bc9aab77ab8-istio-token" pod "20125155-e034-40cd-b7dc-3bc9aab77ab8" (UID: "20125155-e034-40cd-b7dc-3bc9aab77ab8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:33:07.068958 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:07.068884 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/20125155-e034-40cd-b7dc-3bc9aab77ab8-istio-podinfo" pod "20125155-e034-40cd-b7dc-3bc9aab77ab8" (UID: "20125155-e034-40cd-b7dc-3bc9aab77ab8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:33:07.069126 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:07.069108 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20125155-e034-40cd-b7dc-3bc9aab77ab8-kube-api-access-429sb" pod "20125155-e034-40cd-b7dc-3bc9aab77ab8" (UID: "20125155-e034-40cd-b7dc-3bc9aab77ab8"). InnerVolumeSpecName "kube-api-access-429sb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:33:07.082896 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:07.082867 5935 generic.go:356] "Generic (PLEG): container finished" podID="20125155-e034-40cd-b7dc-3bc9aab77ab8" containerID="6d4b0a5e42bf3062afae776def7ab2b0b19a386b44920be78277a4dacedd4666" exitCode=0 Nov 01 20:33:07.083039 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:07.082913 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn" Nov 01 20:33:07.083039 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:07.082918 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn" event={"ID":"20125155-e034-40cd-b7dc-3bc9aab77ab8","Type":"ContainerDied","Data":"6d4b0a5e42bf3062afae776def7ab2b0b19a386b44920be78277a4dacedd4666"} Nov 01 20:33:07.083039 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:07.082973 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn" event={"ID":"20125155-e034-40cd-b7dc-3bc9aab77ab8","Type":"ContainerDied","Data":"4be3b18261fd7e81ccf7d5a8ec385246a75a9624a99e98716cece1934ed31e7f"} Nov 01 20:33:07.083039 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:07.083003 5935 scope.go:122] "RemoveContainer" containerID="6d4b0a5e42bf3062afae776def7ab2b0b19a386b44920be78277a4dacedd4666" Nov 01 20:33:07.093555 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:07.093530 5935 scope.go:122] "RemoveContainer" containerID="6d4b0a5e42bf3062afae776def7ab2b0b19a386b44920be78277a4dacedd4666" Nov 01 20:33:07.093858 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:33:07.093839 5935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d4b0a5e42bf3062afae776def7ab2b0b19a386b44920be78277a4dacedd4666\": container with ID starting with 6d4b0a5e42bf3062afae776def7ab2b0b19a386b44920be78277a4dacedd4666 not found: ID does not exist" containerID="6d4b0a5e42bf3062afae776def7ab2b0b19a386b44920be78277a4dacedd4666" Nov 01 20:33:07.093928 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:07.093872 5935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d4b0a5e42bf3062afae776def7ab2b0b19a386b44920be78277a4dacedd4666"} err="failed to get container status \"6d4b0a5e42bf3062afae776def7ab2b0b19a386b44920be78277a4dacedd4666\": rpc error: code = NotFound desc = could not find container \"6d4b0a5e42bf3062afae776def7ab2b0b19a386b44920be78277a4dacedd4666\": container with ID starting with 6d4b0a5e42bf3062afae776def7ab2b0b19a386b44920be78277a4dacedd4666 not found: ID does not exist" Nov 01 20:33:07.106280 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:07.106253 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn"] Nov 01 20:33:07.110688 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:07.110663 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a6873b-openshift-default-d47989c65-xzckn"] Nov 01 20:33:07.167574 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:07.167540 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-429sb\" (UniqueName: \"kubernetes.io/projected/20125155-e034-40cd-b7dc-3bc9aab77ab8-kube-api-access-429sb\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:33:07.167574 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:07.167567 5935 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20125155-e034-40cd-b7dc-3bc9aab77ab8-istio-envoy\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:33:07.167574 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:07.167580 5935 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20125155-e034-40cd-b7dc-3bc9aab77ab8-istio-token\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:33:07.167823 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:07.167590 5935 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20125155-e034-40cd-b7dc-3bc9aab77ab8-workload-socket\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:33:07.167823 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:07.167600 5935 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20125155-e034-40cd-b7dc-3bc9aab77ab8-istio-podinfo\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:33:07.167823 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:07.167608 5935 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20125155-e034-40cd-b7dc-3bc9aab77ab8-workload-certs\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:33:08.299957 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:08.299892 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0aa61b-5c479c87bc-jsktt" podUID="81191553-1912-45b2-a58a-01fc64d43e23" containerName="payload-pre-processing" containerID="cri-o://3fcac3ac815678d5b9652691475484c922487cc98ced7fa93d6dc3c8ef3b7e4e" gracePeriod=25 Nov 01 20:33:08.352960 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:08.352922 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20125155-e034-40cd-b7dc-3bc9aab77ab8" path="/var/lib/kubelet/pods/20125155-e034-40cd-b7dc-3bc9aab77ab8/volumes" Nov 01 20:33:12.090515 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:12.090458 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-w4-0aa61b-kserve-6674df959d-2lm95" podUID="b7f35703-1219-4dd4-8bbd-631fb7303724" containerName="main" containerID="cri-o://06c8037c5eb2db162c520d833052fd0413f5144056484e4a21e10af771e3c1b2" gracePeriod=45 Nov 01 20:33:12.112856 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:12.112815 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-0aa61b/simulator-w4-0aa61b-kserve-6774594d4b-qmjz8" podUID="09414dc0-7742-48fa-a4b5-571199d99e15" containerName="main" containerID="cri-o://d46de431b842bd003775071eb7b0a1d4cae6f34915597b435d7718ac404aac13" gracePeriod=45 Nov 01 20:33:12.144952 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:12.144906 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-w4-0aa61b-kserve-6674df959d-2lm95" podUID="b7f35703-1219-4dd4-8bbd-631fb7303724" containerName="main" probeResult="failure" output="Get \"https://10.133.0.50:8000/ready\": dial tcp 10.133.0.50:8000: connect: connection refused" Nov 01 20:33:12.149789 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:12.149761 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-0aa61b/simulator-w4-0aa61b-kserve-6774594d4b-qmjz8" podUID="09414dc0-7742-48fa-a4b5-571199d99e15" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": dial tcp 10.133.0.47:8000: connect: connection refused" Nov 01 20:33:12.365422 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:12.365389 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-w4-0aa61b-kserve-6674df959d-2lm95" Nov 01 20:33:12.399719 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:12.399689 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0aa61b/simulator-w4-0aa61b-kserve-6774594d4b-qmjz8" Nov 01 20:33:12.425040 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:12.425001 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7f35703-1219-4dd4-8bbd-631fb7303724-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7f35703-1219-4dd4-8bbd-631fb7303724-tmp-dir\") pod \"b7f35703-1219-4dd4-8bbd-631fb7303724\" (UID: \"b7f35703-1219-4dd4-8bbd-631fb7303724\") " Nov 01 20:33:12.425468 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:12.425423 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7f35703-1219-4dd4-8bbd-631fb7303724-tmp-dir" pod "b7f35703-1219-4dd4-8bbd-631fb7303724" (UID: "b7f35703-1219-4dd4-8bbd-631fb7303724"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:12.425759 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:12.425736 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7f35703-1219-4dd4-8bbd-631fb7303724-model-cache" pod "b7f35703-1219-4dd4-8bbd-631fb7303724" (UID: "b7f35703-1219-4dd4-8bbd-631fb7303724"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:12.425945 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:12.425777 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7f35703-1219-4dd4-8bbd-631fb7303724-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7f35703-1219-4dd4-8bbd-631fb7303724-model-cache\") pod \"b7f35703-1219-4dd4-8bbd-631fb7303724\" (UID: \"b7f35703-1219-4dd4-8bbd-631fb7303724\") " Nov 01 20:33:12.425945 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:12.425915 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7f35703-1219-4dd4-8bbd-631fb7303724-tls-certs\" (UniqueName: \"kubernetes.io/secret/b7f35703-1219-4dd4-8bbd-631fb7303724-tls-certs\") pod \"b7f35703-1219-4dd4-8bbd-631fb7303724\" (UID: \"b7f35703-1219-4dd4-8bbd-631fb7303724\") " Nov 01 20:33:12.426142 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:12.425974 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7f35703-1219-4dd4-8bbd-631fb7303724-home\" (UniqueName: \"kubernetes.io/empty-dir/b7f35703-1219-4dd4-8bbd-631fb7303724-home\") pod \"b7f35703-1219-4dd4-8bbd-631fb7303724\" (UID: \"b7f35703-1219-4dd4-8bbd-631fb7303724\") " Nov 01 20:33:12.426142 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:12.426007 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7f35703-1219-4dd4-8bbd-631fb7303724-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7f35703-1219-4dd4-8bbd-631fb7303724-dshm\") pod \"b7f35703-1219-4dd4-8bbd-631fb7303724\" (UID: \"b7f35703-1219-4dd4-8bbd-631fb7303724\") " Nov 01 20:33:12.426142 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:12.426068 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7f35703-1219-4dd4-8bbd-631fb7303724-kube-api-access-f5z2t\" (UniqueName: \"kubernetes.io/projected/b7f35703-1219-4dd4-8bbd-631fb7303724-kube-api-access-f5z2t\") pod \"b7f35703-1219-4dd4-8bbd-631fb7303724\" (UID: \"b7f35703-1219-4dd4-8bbd-631fb7303724\") " Nov 01 20:33:12.431496 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:12.426636 5935 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7f35703-1219-4dd4-8bbd-631fb7303724-tmp-dir\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:33:12.431496 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:12.426658 5935 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7f35703-1219-4dd4-8bbd-631fb7303724-model-cache\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:33:12.431496 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:12.428790 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7f35703-1219-4dd4-8bbd-631fb7303724-home" pod "b7f35703-1219-4dd4-8bbd-631fb7303724" (UID: "b7f35703-1219-4dd4-8bbd-631fb7303724"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:12.431496 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:12.429317 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7f35703-1219-4dd4-8bbd-631fb7303724-kube-api-access-f5z2t" pod "b7f35703-1219-4dd4-8bbd-631fb7303724" (UID: "b7f35703-1219-4dd4-8bbd-631fb7303724"). InnerVolumeSpecName "kube-api-access-f5z2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:33:12.431496 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:12.430996 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7f35703-1219-4dd4-8bbd-631fb7303724-tls-certs" pod "b7f35703-1219-4dd4-8bbd-631fb7303724" (UID: "b7f35703-1219-4dd4-8bbd-631fb7303724"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:33:12.431496 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:12.431314 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7f35703-1219-4dd4-8bbd-631fb7303724-dshm" pod "b7f35703-1219-4dd4-8bbd-631fb7303724" (UID: "b7f35703-1219-4dd4-8bbd-631fb7303724"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:12.527833 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:12.527789 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09414dc0-7742-48fa-a4b5-571199d99e15-tls-certs\" (UniqueName: \"kubernetes.io/secret/09414dc0-7742-48fa-a4b5-571199d99e15-tls-certs\") pod \"09414dc0-7742-48fa-a4b5-571199d99e15\" (UID: \"09414dc0-7742-48fa-a4b5-571199d99e15\") " Nov 01 20:33:12.528043 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:12.527891 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09414dc0-7742-48fa-a4b5-571199d99e15-dshm\" (UniqueName: \"kubernetes.io/empty-dir/09414dc0-7742-48fa-a4b5-571199d99e15-dshm\") pod \"09414dc0-7742-48fa-a4b5-571199d99e15\" (UID: \"09414dc0-7742-48fa-a4b5-571199d99e15\") " Nov 01 20:33:12.528043 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:12.527970 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09414dc0-7742-48fa-a4b5-571199d99e15-kube-api-access-jcvjm\" (UniqueName: \"kubernetes.io/projected/09414dc0-7742-48fa-a4b5-571199d99e15-kube-api-access-jcvjm\") pod \"09414dc0-7742-48fa-a4b5-571199d99e15\" (UID: \"09414dc0-7742-48fa-a4b5-571199d99e15\") " Nov 01 20:33:12.528043 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:12.528018 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09414dc0-7742-48fa-a4b5-571199d99e15-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09414dc0-7742-48fa-a4b5-571199d99e15-model-cache\") pod \"09414dc0-7742-48fa-a4b5-571199d99e15\" (UID: \"09414dc0-7742-48fa-a4b5-571199d99e15\") " Nov 01 20:33:12.528207 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:12.528077 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09414dc0-7742-48fa-a4b5-571199d99e15-home\" (UniqueName: \"kubernetes.io/empty-dir/09414dc0-7742-48fa-a4b5-571199d99e15-home\") pod \"09414dc0-7742-48fa-a4b5-571199d99e15\" (UID: \"09414dc0-7742-48fa-a4b5-571199d99e15\") " Nov 01 20:33:12.528207 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:12.528113 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09414dc0-7742-48fa-a4b5-571199d99e15-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09414dc0-7742-48fa-a4b5-571199d99e15-tmp-dir\") pod \"09414dc0-7742-48fa-a4b5-571199d99e15\" (UID: \"09414dc0-7742-48fa-a4b5-571199d99e15\") " Nov 01 20:33:12.528424 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:12.528389 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09414dc0-7742-48fa-a4b5-571199d99e15-model-cache" pod "09414dc0-7742-48fa-a4b5-571199d99e15" (UID: "09414dc0-7742-48fa-a4b5-571199d99e15"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:12.528424 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:12.528370 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09414dc0-7742-48fa-a4b5-571199d99e15-home" pod "09414dc0-7742-48fa-a4b5-571199d99e15" (UID: "09414dc0-7742-48fa-a4b5-571199d99e15"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:12.528618 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:12.528430 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09414dc0-7742-48fa-a4b5-571199d99e15-tmp-dir" pod "09414dc0-7742-48fa-a4b5-571199d99e15" (UID: "09414dc0-7742-48fa-a4b5-571199d99e15"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:12.528618 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:12.528546 5935 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7f35703-1219-4dd4-8bbd-631fb7303724-tls-certs\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:33:12.528618 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:12.528559 5935 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09414dc0-7742-48fa-a4b5-571199d99e15-tmp-dir\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:33:12.528618 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:12.528567 5935 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7f35703-1219-4dd4-8bbd-631fb7303724-home\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:33:12.528618 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:12.528575 5935 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7f35703-1219-4dd4-8bbd-631fb7303724-dshm\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:33:12.528618 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:12.528585 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5z2t\" (UniqueName: \"kubernetes.io/projected/b7f35703-1219-4dd4-8bbd-631fb7303724-kube-api-access-f5z2t\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:33:12.528618 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:12.528598 5935 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09414dc0-7742-48fa-a4b5-571199d99e15-model-cache\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:33:12.528618 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:12.528607 5935 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09414dc0-7742-48fa-a4b5-571199d99e15-home\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:33:12.530324 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:12.530278 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09414dc0-7742-48fa-a4b5-571199d99e15-dshm" pod "09414dc0-7742-48fa-a4b5-571199d99e15" (UID: "09414dc0-7742-48fa-a4b5-571199d99e15"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:12.530426 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:12.530409 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09414dc0-7742-48fa-a4b5-571199d99e15-kube-api-access-jcvjm" pod "09414dc0-7742-48fa-a4b5-571199d99e15" (UID: "09414dc0-7742-48fa-a4b5-571199d99e15"). InnerVolumeSpecName "kube-api-access-jcvjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:33:12.530471 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:12.530446 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09414dc0-7742-48fa-a4b5-571199d99e15-tls-certs" pod "09414dc0-7742-48fa-a4b5-571199d99e15" (UID: "09414dc0-7742-48fa-a4b5-571199d99e15"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:33:12.629543 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:12.629458 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcvjm\" (UniqueName: \"kubernetes.io/projected/09414dc0-7742-48fa-a4b5-571199d99e15-kube-api-access-jcvjm\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:33:12.629543 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:12.629488 5935 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09414dc0-7742-48fa-a4b5-571199d99e15-tls-certs\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:33:12.629543 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:12.629498 5935 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09414dc0-7742-48fa-a4b5-571199d99e15-dshm\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:33:13.133395 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:13.133359 5935 generic.go:356] "Generic (PLEG): container finished" podID="b7f35703-1219-4dd4-8bbd-631fb7303724" containerID="06c8037c5eb2db162c520d833052fd0413f5144056484e4a21e10af771e3c1b2" exitCode=0 Nov 01 20:33:13.133876 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:13.133421 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-w4-0aa61b-kserve-6674df959d-2lm95" Nov 01 20:33:13.133876 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:13.133449 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-w4-0aa61b-kserve-6674df959d-2lm95" event={"ID":"b7f35703-1219-4dd4-8bbd-631fb7303724","Type":"ContainerDied","Data":"06c8037c5eb2db162c520d833052fd0413f5144056484e4a21e10af771e3c1b2"} Nov 01 20:33:13.133876 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:13.133495 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0aa61b/distinct-w4-0aa61b-kserve-6674df959d-2lm95" event={"ID":"b7f35703-1219-4dd4-8bbd-631fb7303724","Type":"ContainerDied","Data":"985e8f26fd2c49a598f616f622ecd3f8bffabdff88d04705c1aaf1c3d30fd15f"} Nov 01 20:33:13.133876 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:13.133518 5935 scope.go:122] "RemoveContainer" containerID="06c8037c5eb2db162c520d833052fd0413f5144056484e4a21e10af771e3c1b2" Nov 01 20:33:13.135548 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:13.135514 5935 generic.go:356] "Generic (PLEG): container finished" podID="09414dc0-7742-48fa-a4b5-571199d99e15" containerID="d46de431b842bd003775071eb7b0a1d4cae6f34915597b435d7718ac404aac13" exitCode=0 Nov 01 20:33:13.135664 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:13.135552 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-0aa61b/simulator-w4-0aa61b-kserve-6774594d4b-qmjz8" Nov 01 20:33:13.135664 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:13.135585 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0aa61b/simulator-w4-0aa61b-kserve-6774594d4b-qmjz8" event={"ID":"09414dc0-7742-48fa-a4b5-571199d99e15","Type":"ContainerDied","Data":"d46de431b842bd003775071eb7b0a1d4cae6f34915597b435d7718ac404aac13"} Nov 01 20:33:13.135664 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:13.135639 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-0aa61b/simulator-w4-0aa61b-kserve-6774594d4b-qmjz8" event={"ID":"09414dc0-7742-48fa-a4b5-571199d99e15","Type":"ContainerDied","Data":"1de533698ac5b0d09f1c8258d961c93943893dba8e54f5f7bf0b524a65b4780a"} Nov 01 20:33:13.143421 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:13.143402 5935 scope.go:122] "RemoveContainer" containerID="06c8037c5eb2db162c520d833052fd0413f5144056484e4a21e10af771e3c1b2" Nov 01 20:33:13.143752 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:33:13.143732 5935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06c8037c5eb2db162c520d833052fd0413f5144056484e4a21e10af771e3c1b2\": container with ID starting with 06c8037c5eb2db162c520d833052fd0413f5144056484e4a21e10af771e3c1b2 not found: ID does not exist" containerID="06c8037c5eb2db162c520d833052fd0413f5144056484e4a21e10af771e3c1b2" Nov 01 20:33:13.143837 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:13.143758 5935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06c8037c5eb2db162c520d833052fd0413f5144056484e4a21e10af771e3c1b2"} err="failed to get container status \"06c8037c5eb2db162c520d833052fd0413f5144056484e4a21e10af771e3c1b2\": rpc error: code = NotFound desc = could not find container \"06c8037c5eb2db162c520d833052fd0413f5144056484e4a21e10af771e3c1b2\": container with ID starting with 06c8037c5eb2db162c520d833052fd0413f5144056484e4a21e10af771e3c1b2 not found: ID does not exist" Nov 01 20:33:13.143837 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:13.143776 5935 scope.go:122] "RemoveContainer" containerID="d46de431b842bd003775071eb7b0a1d4cae6f34915597b435d7718ac404aac13" Nov 01 20:33:13.152256 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:13.152237 5935 scope.go:122] "RemoveContainer" containerID="d46de431b842bd003775071eb7b0a1d4cae6f34915597b435d7718ac404aac13" Nov 01 20:33:13.152538 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:33:13.152521 5935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d46de431b842bd003775071eb7b0a1d4cae6f34915597b435d7718ac404aac13\": container with ID starting with d46de431b842bd003775071eb7b0a1d4cae6f34915597b435d7718ac404aac13 not found: ID does not exist" containerID="d46de431b842bd003775071eb7b0a1d4cae6f34915597b435d7718ac404aac13" Nov 01 20:33:13.152589 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:13.152545 5935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d46de431b842bd003775071eb7b0a1d4cae6f34915597b435d7718ac404aac13"} err="failed to get container status \"d46de431b842bd003775071eb7b0a1d4cae6f34915597b435d7718ac404aac13\": rpc error: code = NotFound desc = could not find container \"d46de431b842bd003775071eb7b0a1d4cae6f34915597b435d7718ac404aac13\": container with ID starting with d46de431b842bd003775071eb7b0a1d4cae6f34915597b435d7718ac404aac13 not found: ID does not exist" Nov 01 20:33:13.156186 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:13.156164 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-0aa61b/distinct-w4-0aa61b-kserve-6674df959d-2lm95"] Nov 01 20:33:13.158037 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:13.158013 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-0aa61b/distinct-w4-0aa61b-kserve-6674df959d-2lm95"] Nov 01 20:33:13.167812 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:13.167792 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-0aa61b/simulator-w4-0aa61b-kserve-6774594d4b-qmjz8"] Nov 01 20:33:13.169397 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:13.169378 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-0aa61b/simulator-w4-0aa61b-kserve-6774594d4b-qmjz8"] Nov 01 20:33:14.352261 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:14.352225 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09414dc0-7742-48fa-a4b5-571199d99e15" path="/var/lib/kubelet/pods/09414dc0-7742-48fa-a4b5-571199d99e15/volumes" Nov 01 20:33:14.352746 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:14.352728 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7f35703-1219-4dd4-8bbd-631fb7303724" path="/var/lib/kubelet/pods/b7f35703-1219-4dd4-8bbd-631fb7303724/volumes" Nov 01 20:33:16.703155 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:16.703109 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0aa61b-5c479c87bc-jsktt" podUID="81191553-1912-45b2-a58a-01fc64d43e23" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 20:33:17.716302 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:17.715920 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-0aa61b-74c6dc5659-g7srp"] Nov 01 20:33:17.716302 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:17.716177 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0aa61b-74c6dc5659-g7srp" podUID="d8569d8c-c04e-47dd-895c-2359353049ae" containerName="maas-api" containerID="cri-o://27b809f376135693042692804138141852d0aa1be0995d93b2ca06e0d25bbc61" gracePeriod=30 Nov 01 20:33:18.009159 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:18.009123 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0aa61b-74c6dc5659-g7srp" Nov 01 20:33:18.086237 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:18.086195 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8569d8c-c04e-47dd-895c-2359353049ae-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8569d8c-c04e-47dd-895c-2359353049ae-metrics-certs\") pod \"d8569d8c-c04e-47dd-895c-2359353049ae\" (UID: \"d8569d8c-c04e-47dd-895c-2359353049ae\") " Nov 01 20:33:18.086441 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:18.086304 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8569d8c-c04e-47dd-895c-2359353049ae-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8569d8c-c04e-47dd-895c-2359353049ae-maas-api-tls\") pod \"d8569d8c-c04e-47dd-895c-2359353049ae\" (UID: \"d8569d8c-c04e-47dd-895c-2359353049ae\") " Nov 01 20:33:18.086441 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:18.086361 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8569d8c-c04e-47dd-895c-2359353049ae-kube-api-access-bs4lq\" (UniqueName: \"kubernetes.io/projected/d8569d8c-c04e-47dd-895c-2359353049ae-kube-api-access-bs4lq\") pod \"d8569d8c-c04e-47dd-895c-2359353049ae\" (UID: \"d8569d8c-c04e-47dd-895c-2359353049ae\") " Nov 01 20:33:18.086441 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:18.086435 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8569d8c-c04e-47dd-895c-2359353049ae-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8569d8c-c04e-47dd-895c-2359353049ae-openshift-service-ca\") pod \"d8569d8c-c04e-47dd-895c-2359353049ae\" (UID: \"d8569d8c-c04e-47dd-895c-2359353049ae\") " Nov 01 20:33:18.086893 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:18.086848 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8569d8c-c04e-47dd-895c-2359353049ae-openshift-service-ca" pod "d8569d8c-c04e-47dd-895c-2359353049ae" (UID: "d8569d8c-c04e-47dd-895c-2359353049ae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:33:18.088437 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:18.088412 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8569d8c-c04e-47dd-895c-2359353049ae-maas-api-tls" pod "d8569d8c-c04e-47dd-895c-2359353049ae" (UID: "d8569d8c-c04e-47dd-895c-2359353049ae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:33:18.088534 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:18.088456 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8569d8c-c04e-47dd-895c-2359353049ae-metrics-certs" pod "d8569d8c-c04e-47dd-895c-2359353049ae" (UID: "d8569d8c-c04e-47dd-895c-2359353049ae"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:33:18.088534 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:18.088478 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8569d8c-c04e-47dd-895c-2359353049ae-kube-api-access-bs4lq" pod "d8569d8c-c04e-47dd-895c-2359353049ae" (UID: "d8569d8c-c04e-47dd-895c-2359353049ae"). InnerVolumeSpecName "kube-api-access-bs4lq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:33:18.166526 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:18.166486 5935 generic.go:356] "Generic (PLEG): container finished" podID="d8569d8c-c04e-47dd-895c-2359353049ae" containerID="27b809f376135693042692804138141852d0aa1be0995d93b2ca06e0d25bbc61" exitCode=0 Nov 01 20:33:18.166700 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:18.166536 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0aa61b-74c6dc5659-g7srp" Nov 01 20:33:18.166700 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:18.166569 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0aa61b-74c6dc5659-g7srp" event={"ID":"d8569d8c-c04e-47dd-895c-2359353049ae","Type":"ContainerDied","Data":"27b809f376135693042692804138141852d0aa1be0995d93b2ca06e0d25bbc61"} Nov 01 20:33:18.166700 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:18.166627 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0aa61b-74c6dc5659-g7srp" event={"ID":"d8569d8c-c04e-47dd-895c-2359353049ae","Type":"ContainerDied","Data":"86d7d0ea4ba41dcab25723000764a769052efa7b016fd02e00d2900f775435dd"} Nov 01 20:33:18.166700 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:18.166649 5935 scope.go:122] "RemoveContainer" containerID="27b809f376135693042692804138141852d0aa1be0995d93b2ca06e0d25bbc61" Nov 01 20:33:18.176471 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:18.176407 5935 scope.go:122] "RemoveContainer" containerID="27b809f376135693042692804138141852d0aa1be0995d93b2ca06e0d25bbc61" Nov 01 20:33:18.176714 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:33:18.176691 5935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27b809f376135693042692804138141852d0aa1be0995d93b2ca06e0d25bbc61\": container with ID starting with 27b809f376135693042692804138141852d0aa1be0995d93b2ca06e0d25bbc61 not found: ID does not exist" containerID="27b809f376135693042692804138141852d0aa1be0995d93b2ca06e0d25bbc61" Nov 01 20:33:18.176783 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:18.176721 5935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27b809f376135693042692804138141852d0aa1be0995d93b2ca06e0d25bbc61"} err="failed to get container status \"27b809f376135693042692804138141852d0aa1be0995d93b2ca06e0d25bbc61\": rpc error: code = NotFound desc = could not find container \"27b809f376135693042692804138141852d0aa1be0995d93b2ca06e0d25bbc61\": container with ID starting with 27b809f376135693042692804138141852d0aa1be0995d93b2ca06e0d25bbc61 not found: ID does not exist" Nov 01 20:33:18.187686 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:18.187644 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-0aa61b-74c6dc5659-g7srp"] Nov 01 20:33:18.187899 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:18.187883 5935 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8569d8c-c04e-47dd-895c-2359353049ae-metrics-certs\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:33:18.187947 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:18.187903 5935 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8569d8c-c04e-47dd-895c-2359353049ae-maas-api-tls\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:33:18.187947 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:18.187915 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bs4lq\" (UniqueName: \"kubernetes.io/projected/d8569d8c-c04e-47dd-895c-2359353049ae-kube-api-access-bs4lq\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:33:18.187947 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:18.187925 5935 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8569d8c-c04e-47dd-895c-2359353049ae-openshift-service-ca\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:33:18.191647 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:18.191623 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-0aa61b-74c6dc5659-g7srp"] Nov 01 20:33:18.353127 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:18.353043 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8569d8c-c04e-47dd-895c-2359353049ae" path="/var/lib/kubelet/pods/d8569d8c-c04e-47dd-895c-2359353049ae/volumes" Nov 01 20:33:22.513492 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:22.513459 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e83b4b-86c6f68dcf-zbtwx"] Nov 01 20:33:22.513946 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:22.513678 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e83b4b-86c6f68dcf-zbtwx" podUID="acbd9a98-25e3-4dd9-9f27-2cf274a72c02" containerName="maas-api" containerID="cri-o://ed542080fd41daf8476fe0862169697337b18c520ba465d5fb12f7ab2ec80a08" gracePeriod=30 Nov 01 20:33:22.786478 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:22.786451 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e83b4b-86c6f68dcf-zbtwx" Nov 01 20:33:22.939535 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:22.939502 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acbd9a98-25e3-4dd9-9f27-2cf274a72c02-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acbd9a98-25e3-4dd9-9f27-2cf274a72c02-maas-api-tls\") pod \"acbd9a98-25e3-4dd9-9f27-2cf274a72c02\" (UID: \"acbd9a98-25e3-4dd9-9f27-2cf274a72c02\") " Nov 01 20:33:22.939735 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:22.939548 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acbd9a98-25e3-4dd9-9f27-2cf274a72c02-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acbd9a98-25e3-4dd9-9f27-2cf274a72c02-openshift-service-ca\") pod \"acbd9a98-25e3-4dd9-9f27-2cf274a72c02\" (UID: \"acbd9a98-25e3-4dd9-9f27-2cf274a72c02\") " Nov 01 20:33:22.939735 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:22.939630 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acbd9a98-25e3-4dd9-9f27-2cf274a72c02-kube-api-access-b2pml\" (UniqueName: \"kubernetes.io/projected/acbd9a98-25e3-4dd9-9f27-2cf274a72c02-kube-api-access-b2pml\") pod \"acbd9a98-25e3-4dd9-9f27-2cf274a72c02\" (UID: \"acbd9a98-25e3-4dd9-9f27-2cf274a72c02\") " Nov 01 20:33:22.939735 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:22.939708 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acbd9a98-25e3-4dd9-9f27-2cf274a72c02-metrics-certs\" (UniqueName: \"kubernetes.io/secret/acbd9a98-25e3-4dd9-9f27-2cf274a72c02-metrics-certs\") pod \"acbd9a98-25e3-4dd9-9f27-2cf274a72c02\" (UID: \"acbd9a98-25e3-4dd9-9f27-2cf274a72c02\") " Nov 01 20:33:22.940021 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:22.939989 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acbd9a98-25e3-4dd9-9f27-2cf274a72c02-openshift-service-ca" pod "acbd9a98-25e3-4dd9-9f27-2cf274a72c02" (UID: "acbd9a98-25e3-4dd9-9f27-2cf274a72c02"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:33:22.941706 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:22.941670 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acbd9a98-25e3-4dd9-9f27-2cf274a72c02-maas-api-tls" pod "acbd9a98-25e3-4dd9-9f27-2cf274a72c02" (UID: "acbd9a98-25e3-4dd9-9f27-2cf274a72c02"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:33:22.942240 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:22.942215 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acbd9a98-25e3-4dd9-9f27-2cf274a72c02-metrics-certs" pod "acbd9a98-25e3-4dd9-9f27-2cf274a72c02" (UID: "acbd9a98-25e3-4dd9-9f27-2cf274a72c02"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:33:22.942240 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:22.942226 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acbd9a98-25e3-4dd9-9f27-2cf274a72c02-kube-api-access-b2pml" pod "acbd9a98-25e3-4dd9-9f27-2cf274a72c02" (UID: "acbd9a98-25e3-4dd9-9f27-2cf274a72c02"). InnerVolumeSpecName "kube-api-access-b2pml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:33:23.041077 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:23.040866 5935 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/acbd9a98-25e3-4dd9-9f27-2cf274a72c02-metrics-certs\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:33:23.041077 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:23.040900 5935 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/acbd9a98-25e3-4dd9-9f27-2cf274a72c02-maas-api-tls\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:33:23.041077 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:23.040916 5935 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/acbd9a98-25e3-4dd9-9f27-2cf274a72c02-openshift-service-ca\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:33:23.041077 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:23.040930 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2pml\" (UniqueName: \"kubernetes.io/projected/acbd9a98-25e3-4dd9-9f27-2cf274a72c02-kube-api-access-b2pml\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:33:23.198561 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:23.198521 5935 generic.go:356] "Generic (PLEG): container finished" podID="acbd9a98-25e3-4dd9-9f27-2cf274a72c02" containerID="ed542080fd41daf8476fe0862169697337b18c520ba465d5fb12f7ab2ec80a08" exitCode=0 Nov 01 20:33:23.198734 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:23.198571 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e83b4b-86c6f68dcf-zbtwx" Nov 01 20:33:23.198734 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:23.198617 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e83b4b-86c6f68dcf-zbtwx" event={"ID":"acbd9a98-25e3-4dd9-9f27-2cf274a72c02","Type":"ContainerDied","Data":"ed542080fd41daf8476fe0862169697337b18c520ba465d5fb12f7ab2ec80a08"} Nov 01 20:33:23.198734 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:23.198727 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e83b4b-86c6f68dcf-zbtwx" event={"ID":"acbd9a98-25e3-4dd9-9f27-2cf274a72c02","Type":"ContainerDied","Data":"27e021b97fe1a3e9f1c96cdfee54a9928830dccfcb3d5dced1d674a49b0240f4"} Nov 01 20:33:23.198846 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:23.198765 5935 scope.go:122] "RemoveContainer" containerID="ed542080fd41daf8476fe0862169697337b18c520ba465d5fb12f7ab2ec80a08" Nov 01 20:33:23.209227 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:23.209198 5935 scope.go:122] "RemoveContainer" containerID="ed542080fd41daf8476fe0862169697337b18c520ba465d5fb12f7ab2ec80a08" Nov 01 20:33:23.209582 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:33:23.209561 5935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed542080fd41daf8476fe0862169697337b18c520ba465d5fb12f7ab2ec80a08\": container with ID starting with ed542080fd41daf8476fe0862169697337b18c520ba465d5fb12f7ab2ec80a08 not found: ID does not exist" containerID="ed542080fd41daf8476fe0862169697337b18c520ba465d5fb12f7ab2ec80a08" Nov 01 20:33:23.209666 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:23.209593 5935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed542080fd41daf8476fe0862169697337b18c520ba465d5fb12f7ab2ec80a08"} err="failed to get container status \"ed542080fd41daf8476fe0862169697337b18c520ba465d5fb12f7ab2ec80a08\": rpc error: code = NotFound desc = could not find container \"ed542080fd41daf8476fe0862169697337b18c520ba465d5fb12f7ab2ec80a08\": container with ID starting with ed542080fd41daf8476fe0862169697337b18c520ba465d5fb12f7ab2ec80a08 not found: ID does not exist" Nov 01 20:33:23.220429 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:23.220269 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e83b4b-86c6f68dcf-zbtwx"] Nov 01 20:33:23.225652 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:23.225621 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e83b4b-86c6f68dcf-zbtwx"] Nov 01 20:33:24.352804 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:24.352767 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acbd9a98-25e3-4dd9-9f27-2cf274a72c02" path="/var/lib/kubelet/pods/acbd9a98-25e3-4dd9-9f27-2cf274a72c02/volumes" Nov 01 20:33:26.702700 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:26.702649 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0aa61b-5c479c87bc-jsktt" podUID="81191553-1912-45b2-a58a-01fc64d43e23" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 20:33:28.074222 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:28.074190 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd"] Nov 01 20:33:28.074667 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:28.074632 5935 cpu_manager.go:414] "containerMap: removing container" podUID="c5b342ce-508a-445c-8034-ff83d26a7190" containerName="cleanup" Nov 01 20:33:28.074667 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:28.074645 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5b342ce-508a-445c-8034-ff83d26a7190" containerName="cleanup" Nov 01 20:33:28.074667 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:28.074668 5935 cpu_manager.go:414] "containerMap: removing container" podUID="de43c730-bc16-43b6-8cbb-c32a14622bb7" containerName="payload-processing" Nov 01 20:33:28.074779 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:28.074675 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de43c730-bc16-43b6-8cbb-c32a14622bb7" containerName="payload-processing" Nov 01 20:33:28.074779 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:28.074685 5935 cpu_manager.go:414] "containerMap: removing container" podUID="acbd9a98-25e3-4dd9-9f27-2cf274a72c02" containerName="maas-api" Nov 01 20:33:28.074779 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:28.074690 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acbd9a98-25e3-4dd9-9f27-2cf274a72c02" containerName="maas-api" Nov 01 20:33:28.074779 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:28.074697 5935 cpu_manager.go:414] "containerMap: removing container" podUID="d8569d8c-c04e-47dd-895c-2359353049ae" containerName="maas-api" Nov 01 20:33:28.074779 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:28.074702 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8569d8c-c04e-47dd-895c-2359353049ae" containerName="maas-api" Nov 01 20:33:28.074779 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:28.074711 5935 cpu_manager.go:414] "containerMap: removing container" podUID="20125155-e034-40cd-b7dc-3bc9aab77ab8" containerName="istio-proxy" Nov 01 20:33:28.074779 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:28.074716 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20125155-e034-40cd-b7dc-3bc9aab77ab8" containerName="istio-proxy" Nov 01 20:33:28.074779 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:28.074745 5935 cpu_manager.go:414] "containerMap: removing container" podUID="b7f35703-1219-4dd4-8bbd-631fb7303724" containerName="main" Nov 01 20:33:28.074779 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:28.074750 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7f35703-1219-4dd4-8bbd-631fb7303724" containerName="main" Nov 01 20:33:28.074779 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:28.074763 5935 cpu_manager.go:414] "containerMap: removing container" podUID="6465f212-e931-4b70-8b1a-a1ec15850515" containerName="payload-pre-processing" Nov 01 20:33:28.074779 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:28.074770 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6465f212-e931-4b70-8b1a-a1ec15850515" containerName="payload-pre-processing" Nov 01 20:33:28.076031 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:28.074788 5935 cpu_manager.go:414] "containerMap: removing container" podUID="624bb371-6040-4cd4-a668-a8afc016d132" containerName="payload-processing" Nov 01 20:33:28.076031 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:28.074796 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="624bb371-6040-4cd4-a668-a8afc016d132" containerName="payload-processing" Nov 01 20:33:28.076031 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:28.074807 5935 cpu_manager.go:414] "containerMap: removing container" podUID="09414dc0-7742-48fa-a4b5-571199d99e15" containerName="main" Nov 01 20:33:28.076031 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:28.074814 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09414dc0-7742-48fa-a4b5-571199d99e15" containerName="main" Nov 01 20:33:28.076031 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:28.074903 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="acbd9a98-25e3-4dd9-9f27-2cf274a72c02" containerName="maas-api" Nov 01 20:33:28.076031 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:28.074912 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8569d8c-c04e-47dd-895c-2359353049ae" containerName="maas-api" Nov 01 20:33:28.076031 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:28.074920 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="20125155-e034-40cd-b7dc-3bc9aab77ab8" containerName="istio-proxy" Nov 01 20:33:28.076031 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:28.074927 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7f35703-1219-4dd4-8bbd-631fb7303724" containerName="main" Nov 01 20:33:28.076031 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:28.074933 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="6465f212-e931-4b70-8b1a-a1ec15850515" containerName="payload-pre-processing" Nov 01 20:33:28.076031 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:28.074939 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="624bb371-6040-4cd4-a668-a8afc016d132" containerName="payload-processing" Nov 01 20:33:28.076031 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:28.074944 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="09414dc0-7742-48fa-a4b5-571199d99e15" containerName="main" Nov 01 20:33:28.076031 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:28.074953 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="de43c730-bc16-43b6-8cbb-c32a14622bb7" containerName="payload-processing" Nov 01 20:33:28.078581 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:28.078552 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd" Nov 01 20:33:28.089373 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:28.089345 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd"] Nov 01 20:33:28.192767 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:28.192724 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b4eb318-7389-46d3-a2b0-3f160182503f-istio-envoy\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd\" (UID: \"7b4eb318-7389-46d3-a2b0-3f160182503f\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd" Nov 01 20:33:28.192957 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:28.192798 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7jgr\" (UniqueName: \"kubernetes.io/projected/7b4eb318-7389-46d3-a2b0-3f160182503f-kube-api-access-q7jgr\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd\" (UID: \"7b4eb318-7389-46d3-a2b0-3f160182503f\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd" Nov 01 20:33:28.192957 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:28.192910 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b4eb318-7389-46d3-a2b0-3f160182503f-credential-socket\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd\" (UID: \"7b4eb318-7389-46d3-a2b0-3f160182503f\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd" Nov 01 20:33:28.192957 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:28.192948 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b4eb318-7389-46d3-a2b0-3f160182503f-istio-data\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd\" (UID: \"7b4eb318-7389-46d3-a2b0-3f160182503f\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd" Nov 01 20:33:28.193068 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:28.193021 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b4eb318-7389-46d3-a2b0-3f160182503f-workload-socket\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd\" (UID: \"7b4eb318-7389-46d3-a2b0-3f160182503f\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd" Nov 01 20:33:28.193106 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:28.193065 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b4eb318-7389-46d3-a2b0-3f160182503f-istio-podinfo\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd\" (UID: \"7b4eb318-7389-46d3-a2b0-3f160182503f\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd" Nov 01 20:33:28.193141 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:28.193123 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b4eb318-7389-46d3-a2b0-3f160182503f-workload-certs\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd\" (UID: \"7b4eb318-7389-46d3-a2b0-3f160182503f\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd" Nov 01 20:33:28.193175 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:28.193159 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b4eb318-7389-46d3-a2b0-3f160182503f-istio-token\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd\" (UID: \"7b4eb318-7389-46d3-a2b0-3f160182503f\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd" Nov 01 20:33:28.193209 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:28.193198 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b4eb318-7389-46d3-a2b0-3f160182503f-istiod-ca-cert\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd\" (UID: \"7b4eb318-7389-46d3-a2b0-3f160182503f\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd" Nov 01 20:33:28.294315 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:28.294255 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7jgr\" (UniqueName: \"kubernetes.io/projected/7b4eb318-7389-46d3-a2b0-3f160182503f-kube-api-access-q7jgr\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd\" (UID: \"7b4eb318-7389-46d3-a2b0-3f160182503f\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd" Nov 01 20:33:28.294495 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:28.294352 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b4eb318-7389-46d3-a2b0-3f160182503f-credential-socket\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd\" (UID: \"7b4eb318-7389-46d3-a2b0-3f160182503f\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd" Nov 01 20:33:28.294495 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:28.294389 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b4eb318-7389-46d3-a2b0-3f160182503f-istio-data\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd\" (UID: \"7b4eb318-7389-46d3-a2b0-3f160182503f\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd" Nov 01 20:33:28.294495 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:28.294420 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b4eb318-7389-46d3-a2b0-3f160182503f-workload-socket\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd\" (UID: \"7b4eb318-7389-46d3-a2b0-3f160182503f\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd" Nov 01 20:33:28.294495 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:28.294474 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b4eb318-7389-46d3-a2b0-3f160182503f-istio-podinfo\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd\" (UID: \"7b4eb318-7389-46d3-a2b0-3f160182503f\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd" Nov 01 20:33:28.294660 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:28.294511 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b4eb318-7389-46d3-a2b0-3f160182503f-workload-certs\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd\" (UID: \"7b4eb318-7389-46d3-a2b0-3f160182503f\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd" Nov 01 20:33:28.294660 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:28.294536 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b4eb318-7389-46d3-a2b0-3f160182503f-istio-token\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd\" (UID: \"7b4eb318-7389-46d3-a2b0-3f160182503f\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd" Nov 01 20:33:28.294660 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:28.294570 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b4eb318-7389-46d3-a2b0-3f160182503f-istiod-ca-cert\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd\" (UID: \"7b4eb318-7389-46d3-a2b0-3f160182503f\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd" Nov 01 20:33:28.294660 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:28.294603 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b4eb318-7389-46d3-a2b0-3f160182503f-istio-envoy\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd\" (UID: \"7b4eb318-7389-46d3-a2b0-3f160182503f\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd" Nov 01 20:33:28.294886 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:28.294860 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b4eb318-7389-46d3-a2b0-3f160182503f-istio-data\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd\" (UID: \"7b4eb318-7389-46d3-a2b0-3f160182503f\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd" Nov 01 20:33:28.294944 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:28.294890 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b4eb318-7389-46d3-a2b0-3f160182503f-workload-socket\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd\" (UID: \"7b4eb318-7389-46d3-a2b0-3f160182503f\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd" Nov 01 20:33:28.295087 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:28.295057 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b4eb318-7389-46d3-a2b0-3f160182503f-workload-certs\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd\" (UID: \"7b4eb318-7389-46d3-a2b0-3f160182503f\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd" Nov 01 20:33:28.295087 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:28.295077 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b4eb318-7389-46d3-a2b0-3f160182503f-credential-socket\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd\" (UID: \"7b4eb318-7389-46d3-a2b0-3f160182503f\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd" Nov 01 20:33:28.295325 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:28.295282 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b4eb318-7389-46d3-a2b0-3f160182503f-istiod-ca-cert\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd\" (UID: \"7b4eb318-7389-46d3-a2b0-3f160182503f\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd" Nov 01 20:33:28.296966 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:28.296942 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b4eb318-7389-46d3-a2b0-3f160182503f-istio-envoy\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd\" (UID: \"7b4eb318-7389-46d3-a2b0-3f160182503f\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd" Nov 01 20:33:28.297123 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:28.297105 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b4eb318-7389-46d3-a2b0-3f160182503f-istio-podinfo\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd\" (UID: \"7b4eb318-7389-46d3-a2b0-3f160182503f\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd" Nov 01 20:33:28.302880 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:28.302860 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b4eb318-7389-46d3-a2b0-3f160182503f-istio-token\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd\" (UID: \"7b4eb318-7389-46d3-a2b0-3f160182503f\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd" Nov 01 20:33:28.303041 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:28.303023 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7jgr\" (UniqueName: \"kubernetes.io/projected/7b4eb318-7389-46d3-a2b0-3f160182503f-kube-api-access-q7jgr\") pod \"e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd\" (UID: \"7b4eb318-7389-46d3-a2b0-3f160182503f\") " pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd" Nov 01 20:33:28.394784 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:28.394744 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd" Nov 01 20:33:28.568313 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:28.568261 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0aa61b-5c479c87bc-jsktt" Nov 01 20:33:28.674495 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:28.674433 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd"] Nov 01 20:33:28.676903 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:33:28.676872 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b4eb318_7389_46d3_a2b0_3f160182503f.slice/crio-dfdf76a6644a13117477c5c8def50faa99a8f8d193197c2bb73490f316cb9c7e WatchSource:0}: Error finding container dfdf76a6644a13117477c5c8def50faa99a8f8d193197c2bb73490f316cb9c7e: Status 404 returned error can't find the container with id dfdf76a6644a13117477c5c8def50faa99a8f8d193197c2bb73490f316cb9c7e Nov 01 20:33:28.679118 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:28.679080 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:33:28.679240 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:28.679153 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:33:28.679240 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:28.679184 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:33:28.697651 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:28.697623 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81191553-1912-45b2-a58a-01fc64d43e23-kube-api-access-t6vtn\" (UniqueName: \"kubernetes.io/projected/81191553-1912-45b2-a58a-01fc64d43e23-kube-api-access-t6vtn\") pod \"81191553-1912-45b2-a58a-01fc64d43e23\" (UID: \"81191553-1912-45b2-a58a-01fc64d43e23\") " Nov 01 20:33:28.697782 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:28.697735 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81191553-1912-45b2-a58a-01fc64d43e23-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81191553-1912-45b2-a58a-01fc64d43e23-plugins-config-volume\") pod \"81191553-1912-45b2-a58a-01fc64d43e23\" (UID: \"81191553-1912-45b2-a58a-01fc64d43e23\") " Nov 01 20:33:28.698095 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:28.698073 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81191553-1912-45b2-a58a-01fc64d43e23-plugins-config-volume" pod "81191553-1912-45b2-a58a-01fc64d43e23" (UID: "81191553-1912-45b2-a58a-01fc64d43e23"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:33:28.699559 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:28.699540 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81191553-1912-45b2-a58a-01fc64d43e23-kube-api-access-t6vtn" pod "81191553-1912-45b2-a58a-01fc64d43e23" (UID: "81191553-1912-45b2-a58a-01fc64d43e23"). InnerVolumeSpecName "kube-api-access-t6vtn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:33:28.799683 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:28.799641 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6vtn\" (UniqueName: \"kubernetes.io/projected/81191553-1912-45b2-a58a-01fc64d43e23-kube-api-access-t6vtn\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:33:28.799683 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:28.799686 5935 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81191553-1912-45b2-a58a-01fc64d43e23-plugins-config-volume\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:33:29.232705 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:29.232668 5935 generic.go:356] "Generic (PLEG): container finished" podID="81191553-1912-45b2-a58a-01fc64d43e23" containerID="3fcac3ac815678d5b9652691475484c922487cc98ced7fa93d6dc3c8ef3b7e4e" exitCode=0 Nov 01 20:33:29.233169 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:29.232717 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0aa61b-5c479c87bc-jsktt" Nov 01 20:33:29.233169 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:29.232747 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0aa61b-5c479c87bc-jsktt" event={"ID":"81191553-1912-45b2-a58a-01fc64d43e23","Type":"ContainerDied","Data":"3fcac3ac815678d5b9652691475484c922487cc98ced7fa93d6dc3c8ef3b7e4e"} Nov 01 20:33:29.233169 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:29.232792 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-0aa61b-5c479c87bc-jsktt" event={"ID":"81191553-1912-45b2-a58a-01fc64d43e23","Type":"ContainerDied","Data":"b97f5b832c9290236dd6c97f1b22cd05a67e8bb7f374e9e5d9bb499b6900429f"} Nov 01 20:33:29.233169 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:29.232871 5935 scope.go:122] "RemoveContainer" containerID="3fcac3ac815678d5b9652691475484c922487cc98ced7fa93d6dc3c8ef3b7e4e" Nov 01 20:33:29.234602 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:29.234577 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd" event={"ID":"7b4eb318-7389-46d3-a2b0-3f160182503f","Type":"ContainerStarted","Data":"978a4bf8e32b6356086e3cf4a66e31865d58a1ac5f05f5866456978ad1726575"} Nov 01 20:33:29.234711 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:29.234627 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd" event={"ID":"7b4eb318-7389-46d3-a2b0-3f160182503f","Type":"ContainerStarted","Data":"dfdf76a6644a13117477c5c8def50faa99a8f8d193197c2bb73490f316cb9c7e"} Nov 01 20:33:29.269499 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:29.269465 5935 scope.go:122] "RemoveContainer" containerID="3fcac3ac815678d5b9652691475484c922487cc98ced7fa93d6dc3c8ef3b7e4e" Nov 01 20:33:29.270751 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:33:29.270719 5935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fcac3ac815678d5b9652691475484c922487cc98ced7fa93d6dc3c8ef3b7e4e\": container with ID starting with 3fcac3ac815678d5b9652691475484c922487cc98ced7fa93d6dc3c8ef3b7e4e not found: ID does not exist" containerID="3fcac3ac815678d5b9652691475484c922487cc98ced7fa93d6dc3c8ef3b7e4e" Nov 01 20:33:29.270921 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:29.270758 5935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fcac3ac815678d5b9652691475484c922487cc98ced7fa93d6dc3c8ef3b7e4e"} err="failed to get container status \"3fcac3ac815678d5b9652691475484c922487cc98ced7fa93d6dc3c8ef3b7e4e\": rpc error: code = NotFound desc = could not find container \"3fcac3ac815678d5b9652691475484c922487cc98ced7fa93d6dc3c8ef3b7e4e\": container with ID starting with 3fcac3ac815678d5b9652691475484c922487cc98ced7fa93d6dc3c8ef3b7e4e not found: ID does not exist" Nov 01 20:33:29.271677 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:29.271634 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd" podStartSLOduration=1.271620118 podStartE2EDuration="1.271620118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:33:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:33:29.258888328 +0000 UTC m=+2055.032048840" watchObservedRunningTime="2026-11-01 20:33:29.271620118 +0000 UTC m=+2055.044780667" Nov 01 20:33:29.281651 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:29.281622 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-0aa61b-5c479c87bc-jsktt"] Nov 01 20:33:29.285670 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:29.285646 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-0aa61b-5c479c87bc-jsktt"] Nov 01 20:33:29.395501 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:29.395463 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd" Nov 01 20:33:29.397260 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:29.397231 5935 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Nov 01 20:33:29.397396 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:29.397281 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd" podUID="7b4eb318-7389-46d3-a2b0-3f160182503f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Nov 01 20:33:30.368649 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:30.368607 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81191553-1912-45b2-a58a-01fc64d43e23" path="/var/lib/kubelet/pods/81191553-1912-45b2-a58a-01fc64d43e23/volumes" Nov 01 20:33:30.396097 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:30.396062 5935 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Nov 01 20:33:30.396315 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:30.396126 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd" podUID="7b4eb318-7389-46d3-a2b0-3f160182503f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Nov 01 20:33:31.395696 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:31.395656 5935 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Nov 01 20:33:31.396105 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:31.395729 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd" podUID="7b4eb318-7389-46d3-a2b0-3f160182503f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Nov 01 20:33:32.396154 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:32.396117 5935 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Nov 01 20:33:32.396645 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:32.396171 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd" podUID="7b4eb318-7389-46d3-a2b0-3f160182503f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Nov 01 20:33:33.396247 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:33.396206 5935 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Nov 01 20:33:33.396696 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:33.396300 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd" podUID="7b4eb318-7389-46d3-a2b0-3f160182503f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Nov 01 20:33:34.395805 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:34.395766 5935 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body= Nov 01 20:33:34.395999 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:34.395820 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd" podUID="7b4eb318-7389-46d3-a2b0-3f160182503f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" Nov 01 20:33:35.403845 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:35.403803 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd" Nov 01 20:33:35.404944 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:35.404878 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd" Nov 01 20:33:35.405083 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:35.405030 5935 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd" secret="" err="secret \"e2e-worker-w4-0aa61b-openshift-default-dockercfg-bv7rw\" not found" Nov 01 20:33:35.406041 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:35.406018 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd" Nov 01 20:33:35.415692 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:35.415660 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s"] Nov 01 20:33:35.416135 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:35.416104 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s" podUID="e8bf3d0d-c668-44f1-9238-746e548faba0" containerName="istio-proxy" containerID="cri-o://d9f20bfbfd75bb81796734a93a6ea4e1d1b6b0769209f29666991739058cfdc8" gracePeriod=30 Nov 01 20:33:35.429171 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:35.429132 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd"] Nov 01 20:33:35.845002 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:33:35.844966 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:33:36.144381 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:36.144282 5935 patch_prober.go:29] interesting pod/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 01 20:33:36.144381 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:36.144360 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s" podUID="e8bf3d0d-c668-44f1-9238-746e548faba0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 01 20:33:36.294704 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:36.294668 5935 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd" secret="" err="secret \"e2e-worker-w4-0aa61b-openshift-default-dockercfg-bv7rw\" not found" Nov 01 20:33:37.299624 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:37.299583 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd" podUID="7b4eb318-7389-46d3-a2b0-3f160182503f" containerName="istio-proxy" containerID="cri-o://978a4bf8e32b6356086e3cf4a66e31865d58a1ac5f05f5866456978ad1726575" gracePeriod=30 Nov 01 20:33:39.080181 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:39.080147 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-e38ef4-79f5864c9d-xbg7r"] Nov 01 20:33:39.080711 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:39.080427 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e38ef4-79f5864c9d-xbg7r" podUID="1238788e-4bf8-4a19-8de9-809061a7cb2b" containerName="maas-api" containerID="cri-o://7f888c5280dd1c9f5dd21cc9dd7ab772c23336f638e3ae0fcc8ab92f4dd317ce" gracePeriod=30 Nov 01 20:33:39.312206 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:39.312175 5935 generic.go:356] "Generic (PLEG): container finished" podID="1238788e-4bf8-4a19-8de9-809061a7cb2b" containerID="7f888c5280dd1c9f5dd21cc9dd7ab772c23336f638e3ae0fcc8ab92f4dd317ce" exitCode=0 Nov 01 20:33:39.312398 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:39.312236 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e38ef4-79f5864c9d-xbg7r" event={"ID":"1238788e-4bf8-4a19-8de9-809061a7cb2b","Type":"ContainerDied","Data":"7f888c5280dd1c9f5dd21cc9dd7ab772c23336f638e3ae0fcc8ab92f4dd317ce"} Nov 01 20:33:39.349902 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:39.349876 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e38ef4-79f5864c9d-xbg7r" Nov 01 20:33:39.438309 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:39.438254 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1238788e-4bf8-4a19-8de9-809061a7cb2b-kube-api-access-j99ds\" (UniqueName: \"kubernetes.io/projected/1238788e-4bf8-4a19-8de9-809061a7cb2b-kube-api-access-j99ds\") pod \"1238788e-4bf8-4a19-8de9-809061a7cb2b\" (UID: \"1238788e-4bf8-4a19-8de9-809061a7cb2b\") " Nov 01 20:33:39.438483 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:39.438411 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1238788e-4bf8-4a19-8de9-809061a7cb2b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1238788e-4bf8-4a19-8de9-809061a7cb2b-metrics-certs\") pod \"1238788e-4bf8-4a19-8de9-809061a7cb2b\" (UID: \"1238788e-4bf8-4a19-8de9-809061a7cb2b\") " Nov 01 20:33:39.438526 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:39.438483 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1238788e-4bf8-4a19-8de9-809061a7cb2b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1238788e-4bf8-4a19-8de9-809061a7cb2b-openshift-service-ca\") pod \"1238788e-4bf8-4a19-8de9-809061a7cb2b\" (UID: \"1238788e-4bf8-4a19-8de9-809061a7cb2b\") " Nov 01 20:33:39.438526 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:39.438508 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1238788e-4bf8-4a19-8de9-809061a7cb2b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1238788e-4bf8-4a19-8de9-809061a7cb2b-maas-api-tls\") pod \"1238788e-4bf8-4a19-8de9-809061a7cb2b\" (UID: \"1238788e-4bf8-4a19-8de9-809061a7cb2b\") " Nov 01 20:33:39.438826 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:39.438802 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1238788e-4bf8-4a19-8de9-809061a7cb2b-openshift-service-ca" pod "1238788e-4bf8-4a19-8de9-809061a7cb2b" (UID: "1238788e-4bf8-4a19-8de9-809061a7cb2b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:33:39.440553 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:39.440530 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1238788e-4bf8-4a19-8de9-809061a7cb2b-metrics-certs" pod "1238788e-4bf8-4a19-8de9-809061a7cb2b" (UID: "1238788e-4bf8-4a19-8de9-809061a7cb2b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:33:39.440635 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:39.440561 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1238788e-4bf8-4a19-8de9-809061a7cb2b-maas-api-tls" pod "1238788e-4bf8-4a19-8de9-809061a7cb2b" (UID: "1238788e-4bf8-4a19-8de9-809061a7cb2b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:33:39.440675 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:39.440658 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1238788e-4bf8-4a19-8de9-809061a7cb2b-kube-api-access-j99ds" pod "1238788e-4bf8-4a19-8de9-809061a7cb2b" (UID: "1238788e-4bf8-4a19-8de9-809061a7cb2b"). InnerVolumeSpecName "kube-api-access-j99ds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:33:39.539539 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:39.539505 5935 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1238788e-4bf8-4a19-8de9-809061a7cb2b-metrics-certs\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:33:39.539539 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:39.539535 5935 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1238788e-4bf8-4a19-8de9-809061a7cb2b-openshift-service-ca\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:33:39.539539 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:39.539544 5935 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1238788e-4bf8-4a19-8de9-809061a7cb2b-maas-api-tls\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:33:39.539793 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:39.539553 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j99ds\" (UniqueName: \"kubernetes.io/projected/1238788e-4bf8-4a19-8de9-809061a7cb2b-kube-api-access-j99ds\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:33:40.318955 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:40.318917 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e38ef4-79f5864c9d-xbg7r" Nov 01 20:33:40.319477 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:40.318923 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-e38ef4-79f5864c9d-xbg7r" event={"ID":"1238788e-4bf8-4a19-8de9-809061a7cb2b","Type":"ContainerDied","Data":"cd4b77b6616136f830ec753da3bbeaae75699233bf28c82e17a30a34e9ddfb90"} Nov 01 20:33:40.319477 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:40.319098 5935 scope.go:122] "RemoveContainer" containerID="7f888c5280dd1c9f5dd21cc9dd7ab772c23336f638e3ae0fcc8ab92f4dd317ce" Nov 01 20:33:40.340380 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:40.340339 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-e38ef4-79f5864c9d-xbg7r"] Nov 01 20:33:40.343296 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:40.343256 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-e38ef4-79f5864c9d-xbg7r"] Nov 01 20:33:40.352520 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:40.352494 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1238788e-4bf8-4a19-8de9-809061a7cb2b" path="/var/lib/kubelet/pods/1238788e-4bf8-4a19-8de9-809061a7cb2b/volumes" Nov 01 20:33:40.679762 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:40.679735 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s" Nov 01 20:33:40.753013 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:40.752984 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8bf3d0d-c668-44f1-9238-746e548faba0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8bf3d0d-c668-44f1-9238-746e548faba0-istio-data\") pod \"e8bf3d0d-c668-44f1-9238-746e548faba0\" (UID: \"e8bf3d0d-c668-44f1-9238-746e548faba0\") " Nov 01 20:33:40.753173 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:40.753046 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8bf3d0d-c668-44f1-9238-746e548faba0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bf3d0d-c668-44f1-9238-746e548faba0-credential-socket\") pod \"e8bf3d0d-c668-44f1-9238-746e548faba0\" (UID: \"e8bf3d0d-c668-44f1-9238-746e548faba0\") " Nov 01 20:33:40.753173 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:40.753095 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8bf3d0d-c668-44f1-9238-746e548faba0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bf3d0d-c668-44f1-9238-746e548faba0-workload-socket\") pod \"e8bf3d0d-c668-44f1-9238-746e548faba0\" (UID: \"e8bf3d0d-c668-44f1-9238-746e548faba0\") " Nov 01 20:33:40.753173 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:40.753127 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e8bf3d0d-c668-44f1-9238-746e548faba0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8bf3d0d-c668-44f1-9238-746e548faba0-istio-podinfo\") pod \"e8bf3d0d-c668-44f1-9238-746e548faba0\" (UID: \"e8bf3d0d-c668-44f1-9238-746e548faba0\") " Nov 01 20:33:40.753361 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:40.753174 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8bf3d0d-c668-44f1-9238-746e548faba0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8bf3d0d-c668-44f1-9238-746e548faba0-istiod-ca-cert\") pod \"e8bf3d0d-c668-44f1-9238-746e548faba0\" (UID: \"e8bf3d0d-c668-44f1-9238-746e548faba0\") " Nov 01 20:33:40.753361 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:40.753205 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8bf3d0d-c668-44f1-9238-746e548faba0-kube-api-access-h6vzt\" (UniqueName: \"kubernetes.io/projected/e8bf3d0d-c668-44f1-9238-746e548faba0-kube-api-access-h6vzt\") pod \"e8bf3d0d-c668-44f1-9238-746e548faba0\" (UID: \"e8bf3d0d-c668-44f1-9238-746e548faba0\") " Nov 01 20:33:40.753361 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:40.753233 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8bf3d0d-c668-44f1-9238-746e548faba0-istio-token\" (UniqueName: \"kubernetes.io/projected/e8bf3d0d-c668-44f1-9238-746e548faba0-istio-token\") pod \"e8bf3d0d-c668-44f1-9238-746e548faba0\" (UID: \"e8bf3d0d-c668-44f1-9238-746e548faba0\") " Nov 01 20:33:40.753361 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:40.753281 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8bf3d0d-c668-44f1-9238-746e548faba0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8bf3d0d-c668-44f1-9238-746e548faba0-istio-envoy\") pod \"e8bf3d0d-c668-44f1-9238-746e548faba0\" (UID: \"e8bf3d0d-c668-44f1-9238-746e548faba0\") " Nov 01 20:33:40.753361 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:40.753345 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8bf3d0d-c668-44f1-9238-746e548faba0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8bf3d0d-c668-44f1-9238-746e548faba0-workload-certs\") pod \"e8bf3d0d-c668-44f1-9238-746e548faba0\" (UID: \"e8bf3d0d-c668-44f1-9238-746e548faba0\") " Nov 01 20:33:40.753626 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:40.753349 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8bf3d0d-c668-44f1-9238-746e548faba0-istio-data" pod "e8bf3d0d-c668-44f1-9238-746e548faba0" (UID: "e8bf3d0d-c668-44f1-9238-746e548faba0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:40.753626 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:40.753420 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8bf3d0d-c668-44f1-9238-746e548faba0-credential-socket" pod "e8bf3d0d-c668-44f1-9238-746e548faba0" (UID: "e8bf3d0d-c668-44f1-9238-746e548faba0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:40.753737 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:40.753695 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8bf3d0d-c668-44f1-9238-746e548faba0-workload-socket" pod "e8bf3d0d-c668-44f1-9238-746e548faba0" (UID: "e8bf3d0d-c668-44f1-9238-746e548faba0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:40.753737 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:40.753702 5935 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8bf3d0d-c668-44f1-9238-746e548faba0-istio-data\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:33:40.753854 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:40.753743 5935 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bf3d0d-c668-44f1-9238-746e548faba0-credential-socket\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:33:40.753854 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:40.753753 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8bf3d0d-c668-44f1-9238-746e548faba0-istiod-ca-cert" pod "e8bf3d0d-c668-44f1-9238-746e548faba0" (UID: "e8bf3d0d-c668-44f1-9238-746e548faba0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:33:40.753929 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:40.753845 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8bf3d0d-c668-44f1-9238-746e548faba0-workload-certs" pod "e8bf3d0d-c668-44f1-9238-746e548faba0" (UID: "e8bf3d0d-c668-44f1-9238-746e548faba0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:40.755451 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:40.755409 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e8bf3d0d-c668-44f1-9238-746e548faba0-istio-podinfo" pod "e8bf3d0d-c668-44f1-9238-746e548faba0" (UID: "e8bf3d0d-c668-44f1-9238-746e548faba0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:33:40.755583 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:40.755506 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8bf3d0d-c668-44f1-9238-746e548faba0-kube-api-access-h6vzt" pod "e8bf3d0d-c668-44f1-9238-746e548faba0" (UID: "e8bf3d0d-c668-44f1-9238-746e548faba0"). InnerVolumeSpecName "kube-api-access-h6vzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:33:40.755871 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:40.755852 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8bf3d0d-c668-44f1-9238-746e548faba0-istio-envoy" pod "e8bf3d0d-c668-44f1-9238-746e548faba0" (UID: "e8bf3d0d-c668-44f1-9238-746e548faba0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:40.755923 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:40.755879 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8bf3d0d-c668-44f1-9238-746e548faba0-istio-token" pod "e8bf3d0d-c668-44f1-9238-746e548faba0" (UID: "e8bf3d0d-c668-44f1-9238-746e548faba0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:33:40.854677 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:40.854593 5935 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8bf3d0d-c668-44f1-9238-746e548faba0-istio-podinfo\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:33:40.854677 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:40.854627 5935 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8bf3d0d-c668-44f1-9238-746e548faba0-istiod-ca-cert\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:33:40.854677 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:40.854641 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6vzt\" (UniqueName: \"kubernetes.io/projected/e8bf3d0d-c668-44f1-9238-746e548faba0-kube-api-access-h6vzt\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:33:40.854677 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:40.854652 5935 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8bf3d0d-c668-44f1-9238-746e548faba0-istio-token\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:33:40.854677 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:40.854661 5935 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8bf3d0d-c668-44f1-9238-746e548faba0-istio-envoy\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:33:40.854677 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:40.854669 5935 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8bf3d0d-c668-44f1-9238-746e548faba0-workload-certs\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:33:40.854677 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:40.854678 5935 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8bf3d0d-c668-44f1-9238-746e548faba0-workload-socket\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:33:41.325993 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:41.325958 5935 generic.go:356] "Generic (PLEG): container finished" podID="e8bf3d0d-c668-44f1-9238-746e548faba0" containerID="d9f20bfbfd75bb81796734a93a6ea4e1d1b6b0769209f29666991739058cfdc8" exitCode=0 Nov 01 20:33:41.326403 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:41.326006 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s" Nov 01 20:33:41.326403 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:41.326047 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s" event={"ID":"e8bf3d0d-c668-44f1-9238-746e548faba0","Type":"ContainerDied","Data":"d9f20bfbfd75bb81796734a93a6ea4e1d1b6b0769209f29666991739058cfdc8"} Nov 01 20:33:41.326403 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:41.326103 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s" event={"ID":"e8bf3d0d-c668-44f1-9238-746e548faba0","Type":"ContainerDied","Data":"074d62fd641901f6faf175995083fbeb89daee707c1dd5568aeb4546feb8a899"} Nov 01 20:33:41.326403 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:41.326126 5935 scope.go:122] "RemoveContainer" containerID="d9f20bfbfd75bb81796734a93a6ea4e1d1b6b0769209f29666991739058cfdc8" Nov 01 20:33:41.336829 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:41.336807 5935 scope.go:122] "RemoveContainer" containerID="d9f20bfbfd75bb81796734a93a6ea4e1d1b6b0769209f29666991739058cfdc8" Nov 01 20:33:41.337117 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:33:41.337098 5935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9f20bfbfd75bb81796734a93a6ea4e1d1b6b0769209f29666991739058cfdc8\": container with ID starting with d9f20bfbfd75bb81796734a93a6ea4e1d1b6b0769209f29666991739058cfdc8 not found: ID does not exist" containerID="d9f20bfbfd75bb81796734a93a6ea4e1d1b6b0769209f29666991739058cfdc8" Nov 01 20:33:41.337158 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:41.337135 5935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9f20bfbfd75bb81796734a93a6ea4e1d1b6b0769209f29666991739058cfdc8"} err="failed to get container status \"d9f20bfbfd75bb81796734a93a6ea4e1d1b6b0769209f29666991739058cfdc8\": rpc error: code = NotFound desc = could not find container \"d9f20bfbfd75bb81796734a93a6ea4e1d1b6b0769209f29666991739058cfdc8\": container with ID starting with d9f20bfbfd75bb81796734a93a6ea4e1d1b6b0769209f29666991739058cfdc8 not found: ID does not exist" Nov 01 20:33:41.352726 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:41.352698 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s"] Nov 01 20:33:41.356595 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:41.356573 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-z849s"] Nov 01 20:33:42.352155 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:42.352116 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8bf3d0d-c668-44f1-9238-746e548faba0" path="/var/lib/kubelet/pods/e8bf3d0d-c668-44f1-9238-746e548faba0/volumes" Nov 01 20:33:42.562489 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:42.562422 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd" Nov 01 20:33:42.674353 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:42.674316 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b4eb318-7389-46d3-a2b0-3f160182503f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b4eb318-7389-46d3-a2b0-3f160182503f-credential-socket\") pod \"7b4eb318-7389-46d3-a2b0-3f160182503f\" (UID: \"7b4eb318-7389-46d3-a2b0-3f160182503f\") " Nov 01 20:33:42.674353 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:42.674356 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b4eb318-7389-46d3-a2b0-3f160182503f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b4eb318-7389-46d3-a2b0-3f160182503f-istio-data\") pod \"7b4eb318-7389-46d3-a2b0-3f160182503f\" (UID: \"7b4eb318-7389-46d3-a2b0-3f160182503f\") " Nov 01 20:33:42.674605 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:42.674381 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b4eb318-7389-46d3-a2b0-3f160182503f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b4eb318-7389-46d3-a2b0-3f160182503f-workload-certs\") pod \"7b4eb318-7389-46d3-a2b0-3f160182503f\" (UID: \"7b4eb318-7389-46d3-a2b0-3f160182503f\") " Nov 01 20:33:42.674605 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:42.674413 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b4eb318-7389-46d3-a2b0-3f160182503f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b4eb318-7389-46d3-a2b0-3f160182503f-istio-envoy\") pod \"7b4eb318-7389-46d3-a2b0-3f160182503f\" (UID: \"7b4eb318-7389-46d3-a2b0-3f160182503f\") " Nov 01 20:33:42.674605 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:42.674450 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b4eb318-7389-46d3-a2b0-3f160182503f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b4eb318-7389-46d3-a2b0-3f160182503f-workload-socket\") pod \"7b4eb318-7389-46d3-a2b0-3f160182503f\" (UID: \"7b4eb318-7389-46d3-a2b0-3f160182503f\") " Nov 01 20:33:42.674605 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:42.674469 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b4eb318-7389-46d3-a2b0-3f160182503f-istio-token\" (UniqueName: \"kubernetes.io/projected/7b4eb318-7389-46d3-a2b0-3f160182503f-istio-token\") pod \"7b4eb318-7389-46d3-a2b0-3f160182503f\" (UID: \"7b4eb318-7389-46d3-a2b0-3f160182503f\") " Nov 01 20:33:42.674605 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:42.674567 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b4eb318-7389-46d3-a2b0-3f160182503f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b4eb318-7389-46d3-a2b0-3f160182503f-istiod-ca-cert\") pod \"7b4eb318-7389-46d3-a2b0-3f160182503f\" (UID: \"7b4eb318-7389-46d3-a2b0-3f160182503f\") " Nov 01 20:33:42.674605 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:42.674594 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7b4eb318-7389-46d3-a2b0-3f160182503f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b4eb318-7389-46d3-a2b0-3f160182503f-istio-podinfo\") pod \"7b4eb318-7389-46d3-a2b0-3f160182503f\" (UID: \"7b4eb318-7389-46d3-a2b0-3f160182503f\") " Nov 01 20:33:42.674886 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:42.674635 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b4eb318-7389-46d3-a2b0-3f160182503f-kube-api-access-q7jgr\" (UniqueName: \"kubernetes.io/projected/7b4eb318-7389-46d3-a2b0-3f160182503f-kube-api-access-q7jgr\") pod \"7b4eb318-7389-46d3-a2b0-3f160182503f\" (UID: \"7b4eb318-7389-46d3-a2b0-3f160182503f\") " Nov 01 20:33:42.674886 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:42.674655 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b4eb318-7389-46d3-a2b0-3f160182503f-credential-socket" pod "7b4eb318-7389-46d3-a2b0-3f160182503f" (UID: "7b4eb318-7389-46d3-a2b0-3f160182503f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:42.674886 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:42.674679 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b4eb318-7389-46d3-a2b0-3f160182503f-istio-data" pod "7b4eb318-7389-46d3-a2b0-3f160182503f" (UID: "7b4eb318-7389-46d3-a2b0-3f160182503f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:42.674886 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:42.674758 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b4eb318-7389-46d3-a2b0-3f160182503f-workload-certs" pod "7b4eb318-7389-46d3-a2b0-3f160182503f" (UID: "7b4eb318-7389-46d3-a2b0-3f160182503f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:42.674886 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:42.674783 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b4eb318-7389-46d3-a2b0-3f160182503f-workload-socket" pod "7b4eb318-7389-46d3-a2b0-3f160182503f" (UID: "7b4eb318-7389-46d3-a2b0-3f160182503f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:42.675141 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:42.674982 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b4eb318-7389-46d3-a2b0-3f160182503f-istiod-ca-cert" pod "7b4eb318-7389-46d3-a2b0-3f160182503f" (UID: "7b4eb318-7389-46d3-a2b0-3f160182503f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:33:42.675141 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:42.675056 5935 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b4eb318-7389-46d3-a2b0-3f160182503f-workload-socket\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:33:42.675141 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:42.675080 5935 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b4eb318-7389-46d3-a2b0-3f160182503f-credential-socket\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:33:42.675141 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:42.675095 5935 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b4eb318-7389-46d3-a2b0-3f160182503f-istio-data\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:33:42.675141 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:42.675109 5935 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b4eb318-7389-46d3-a2b0-3f160182503f-workload-certs\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:33:42.676891 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:42.676862 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b4eb318-7389-46d3-a2b0-3f160182503f-istio-podinfo" pod "7b4eb318-7389-46d3-a2b0-3f160182503f" (UID: "7b4eb318-7389-46d3-a2b0-3f160182503f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:33:42.677023 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:42.676890 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b4eb318-7389-46d3-a2b0-3f160182503f-istio-token" pod "7b4eb318-7389-46d3-a2b0-3f160182503f" (UID: "7b4eb318-7389-46d3-a2b0-3f160182503f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:33:42.677023 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:42.676907 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b4eb318-7389-46d3-a2b0-3f160182503f-kube-api-access-q7jgr" pod "7b4eb318-7389-46d3-a2b0-3f160182503f" (UID: "7b4eb318-7389-46d3-a2b0-3f160182503f"). InnerVolumeSpecName "kube-api-access-q7jgr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:33:42.677023 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:42.676869 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b4eb318-7389-46d3-a2b0-3f160182503f-istio-envoy" pod "7b4eb318-7389-46d3-a2b0-3f160182503f" (UID: "7b4eb318-7389-46d3-a2b0-3f160182503f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:33:42.775645 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:42.775611 5935 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b4eb318-7389-46d3-a2b0-3f160182503f-istio-token\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:33:42.775645 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:42.775638 5935 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b4eb318-7389-46d3-a2b0-3f160182503f-istiod-ca-cert\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:33:42.775645 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:42.775648 5935 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b4eb318-7389-46d3-a2b0-3f160182503f-istio-podinfo\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:33:42.775887 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:42.775657 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7jgr\" (UniqueName: \"kubernetes.io/projected/7b4eb318-7389-46d3-a2b0-3f160182503f-kube-api-access-q7jgr\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:33:42.775887 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:42.775668 5935 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b4eb318-7389-46d3-a2b0-3f160182503f-istio-envoy\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:33:43.339193 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:43.339159 5935 generic.go:356] "Generic (PLEG): container finished" podID="7b4eb318-7389-46d3-a2b0-3f160182503f" containerID="978a4bf8e32b6356086e3cf4a66e31865d58a1ac5f05f5866456978ad1726575" exitCode=0 Nov 01 20:33:43.339379 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:43.339210 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd" Nov 01 20:33:43.339379 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:43.339249 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd" event={"ID":"7b4eb318-7389-46d3-a2b0-3f160182503f","Type":"ContainerDied","Data":"978a4bf8e32b6356086e3cf4a66e31865d58a1ac5f05f5866456978ad1726575"} Nov 01 20:33:43.339379 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:43.339331 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd" event={"ID":"7b4eb318-7389-46d3-a2b0-3f160182503f","Type":"ContainerDied","Data":"dfdf76a6644a13117477c5c8def50faa99a8f8d193197c2bb73490f316cb9c7e"} Nov 01 20:33:43.339379 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:43.339362 5935 scope.go:122] "RemoveContainer" containerID="978a4bf8e32b6356086e3cf4a66e31865d58a1ac5f05f5866456978ad1726575" Nov 01 20:33:43.349638 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:43.349613 5935 scope.go:122] "RemoveContainer" containerID="978a4bf8e32b6356086e3cf4a66e31865d58a1ac5f05f5866456978ad1726575" Nov 01 20:33:43.349956 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:33:43.349936 5935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"978a4bf8e32b6356086e3cf4a66e31865d58a1ac5f05f5866456978ad1726575\": container with ID starting with 978a4bf8e32b6356086e3cf4a66e31865d58a1ac5f05f5866456978ad1726575 not found: ID does not exist" containerID="978a4bf8e32b6356086e3cf4a66e31865d58a1ac5f05f5866456978ad1726575" Nov 01 20:33:43.350021 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:43.349964 5935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"978a4bf8e32b6356086e3cf4a66e31865d58a1ac5f05f5866456978ad1726575"} err="failed to get container status \"978a4bf8e32b6356086e3cf4a66e31865d58a1ac5f05f5866456978ad1726575\": rpc error: code = NotFound desc = could not find container \"978a4bf8e32b6356086e3cf4a66e31865d58a1ac5f05f5866456978ad1726575\": container with ID starting with 978a4bf8e32b6356086e3cf4a66e31865d58a1ac5f05f5866456978ad1726575 not found: ID does not exist" Nov 01 20:33:43.363581 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:43.363548 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd"] Nov 01 20:33:43.367193 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:43.367164 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0aa61b-openshift-default-677b6f9d9f-4h9hd"] Nov 01 20:33:44.352676 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:44.352641 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b4eb318-7389-46d3-a2b0-3f160182503f" path="/var/lib/kubelet/pods/7b4eb318-7389-46d3-a2b0-3f160182503f/volumes" Nov 01 20:33:51.374067 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:51.374022 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-5370bb-648c6878bc-j99qx"] Nov 01 20:33:51.386897 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:51.386863 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-5370bb-6c5d87878c-kng8d"] Nov 01 20:33:51.894737 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:51.894701 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-5370bb-6f959ddc8c-l67b6"] Nov 01 20:33:51.894960 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:51.894921 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5370bb-6f959ddc8c-l67b6" podUID="32159d48-5e0e-4841-a448-c57d21ab523f" containerName="maas-api" containerID="cri-o://aad073906bb2c4b404c8045eb809f4a5484c4f14eddca678c9b20cd5188dc05d" gracePeriod=30 Nov 01 20:33:52.156092 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:52.156029 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5370bb-6f959ddc8c-l67b6" Nov 01 20:33:52.267036 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:52.267004 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32159d48-5e0e-4841-a448-c57d21ab523f-kube-api-access-s2wxd\" (UniqueName: \"kubernetes.io/projected/32159d48-5e0e-4841-a448-c57d21ab523f-kube-api-access-s2wxd\") pod \"32159d48-5e0e-4841-a448-c57d21ab523f\" (UID: \"32159d48-5e0e-4841-a448-c57d21ab523f\") " Nov 01 20:33:52.267036 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:52.267053 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32159d48-5e0e-4841-a448-c57d21ab523f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32159d48-5e0e-4841-a448-c57d21ab523f-openshift-service-ca\") pod \"32159d48-5e0e-4841-a448-c57d21ab523f\" (UID: \"32159d48-5e0e-4841-a448-c57d21ab523f\") " Nov 01 20:33:52.267346 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:52.267121 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32159d48-5e0e-4841-a448-c57d21ab523f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32159d48-5e0e-4841-a448-c57d21ab523f-maas-api-tls\") pod \"32159d48-5e0e-4841-a448-c57d21ab523f\" (UID: \"32159d48-5e0e-4841-a448-c57d21ab523f\") " Nov 01 20:33:52.267346 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:52.267170 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32159d48-5e0e-4841-a448-c57d21ab523f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/32159d48-5e0e-4841-a448-c57d21ab523f-metrics-certs\") pod \"32159d48-5e0e-4841-a448-c57d21ab523f\" (UID: \"32159d48-5e0e-4841-a448-c57d21ab523f\") " Nov 01 20:33:52.267503 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:52.267473 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32159d48-5e0e-4841-a448-c57d21ab523f-openshift-service-ca" pod "32159d48-5e0e-4841-a448-c57d21ab523f" (UID: "32159d48-5e0e-4841-a448-c57d21ab523f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:33:52.269367 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:52.269333 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32159d48-5e0e-4841-a448-c57d21ab523f-maas-api-tls" pod "32159d48-5e0e-4841-a448-c57d21ab523f" (UID: "32159d48-5e0e-4841-a448-c57d21ab523f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:33:52.269367 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:52.269355 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32159d48-5e0e-4841-a448-c57d21ab523f-metrics-certs" pod "32159d48-5e0e-4841-a448-c57d21ab523f" (UID: "32159d48-5e0e-4841-a448-c57d21ab523f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:33:52.269540 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:52.269524 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32159d48-5e0e-4841-a448-c57d21ab523f-kube-api-access-s2wxd" pod "32159d48-5e0e-4841-a448-c57d21ab523f" (UID: "32159d48-5e0e-4841-a448-c57d21ab523f"). InnerVolumeSpecName "kube-api-access-s2wxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:33:52.368059 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:52.368033 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2wxd\" (UniqueName: \"kubernetes.io/projected/32159d48-5e0e-4841-a448-c57d21ab523f-kube-api-access-s2wxd\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:33:52.368059 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:52.368059 5935 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32159d48-5e0e-4841-a448-c57d21ab523f-openshift-service-ca\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:33:52.368226 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:52.368070 5935 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32159d48-5e0e-4841-a448-c57d21ab523f-maas-api-tls\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:33:52.368226 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:52.368079 5935 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/32159d48-5e0e-4841-a448-c57d21ab523f-metrics-certs\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:33:52.394102 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:52.394067 5935 generic.go:356] "Generic (PLEG): container finished" podID="32159d48-5e0e-4841-a448-c57d21ab523f" containerID="aad073906bb2c4b404c8045eb809f4a5484c4f14eddca678c9b20cd5188dc05d" exitCode=0 Nov 01 20:33:52.394727 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:52.394129 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5370bb-6f959ddc8c-l67b6" Nov 01 20:33:52.394727 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:52.394148 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5370bb-6f959ddc8c-l67b6" event={"ID":"32159d48-5e0e-4841-a448-c57d21ab523f","Type":"ContainerDied","Data":"aad073906bb2c4b404c8045eb809f4a5484c4f14eddca678c9b20cd5188dc05d"} Nov 01 20:33:52.394727 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:52.394226 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-5370bb-6f959ddc8c-l67b6" event={"ID":"32159d48-5e0e-4841-a448-c57d21ab523f","Type":"ContainerDied","Data":"da86dede87a3d22867878eb5f2baf095304dfc0f35a3e318ae22945d6fb897e5"} Nov 01 20:33:52.394727 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:52.394248 5935 scope.go:122] "RemoveContainer" containerID="aad073906bb2c4b404c8045eb809f4a5484c4f14eddca678c9b20cd5188dc05d" Nov 01 20:33:52.403942 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:52.403923 5935 scope.go:122] "RemoveContainer" containerID="aad073906bb2c4b404c8045eb809f4a5484c4f14eddca678c9b20cd5188dc05d" Nov 01 20:33:52.404219 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:33:52.404196 5935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aad073906bb2c4b404c8045eb809f4a5484c4f14eddca678c9b20cd5188dc05d\": container with ID starting with aad073906bb2c4b404c8045eb809f4a5484c4f14eddca678c9b20cd5188dc05d not found: ID does not exist" containerID="aad073906bb2c4b404c8045eb809f4a5484c4f14eddca678c9b20cd5188dc05d" Nov 01 20:33:52.404326 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:52.404226 5935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aad073906bb2c4b404c8045eb809f4a5484c4f14eddca678c9b20cd5188dc05d"} err="failed to get container status \"aad073906bb2c4b404c8045eb809f4a5484c4f14eddca678c9b20cd5188dc05d\": rpc error: code = NotFound desc = could not find container \"aad073906bb2c4b404c8045eb809f4a5484c4f14eddca678c9b20cd5188dc05d\": container with ID starting with aad073906bb2c4b404c8045eb809f4a5484c4f14eddca678c9b20cd5188dc05d not found: ID does not exist" Nov 01 20:33:52.409675 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:52.409618 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-5370bb-6f959ddc8c-l67b6"] Nov 01 20:33:52.413428 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:52.413407 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-5370bb-6f959ddc8c-l67b6"] Nov 01 20:33:54.352386 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:54.352349 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32159d48-5e0e-4841-a448-c57d21ab523f" path="/var/lib/kubelet/pods/32159d48-5e0e-4841-a448-c57d21ab523f/volumes" Nov 01 20:33:56.374747 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:56.374649 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-5370bb-648c6878bc-j99qx" podUID="04b7431a-dbc2-47b6-92b3-da82b44a1b8f" containerName="payload-processing" containerID="cri-o://da1b3779c285509a75104127f4259c714290350f3c184984032c1f4b3285cc74" gracePeriod=25 Nov 01 20:33:56.388188 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:56.388158 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5370bb-6c5d87878c-kng8d" podUID="4b4d32cc-fa47-4b9d-9e8b-c39dfd020196" containerName="payload-pre-processing" containerID="cri-o://0d09088a310336b869185a4926d8b57aad71b987620f5e0eddf5d416c9ac2c90" gracePeriod=25 Nov 01 20:33:56.635389 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:56.635318 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-5370bb-648c6878bc-j99qx" Nov 01 20:33:56.712652 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:56.712616 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04b7431a-dbc2-47b6-92b3-da82b44a1b8f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04b7431a-dbc2-47b6-92b3-da82b44a1b8f-plugins-config-volume\") pod \"04b7431a-dbc2-47b6-92b3-da82b44a1b8f\" (UID: \"04b7431a-dbc2-47b6-92b3-da82b44a1b8f\") " Nov 01 20:33:56.712841 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:56.712742 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04b7431a-dbc2-47b6-92b3-da82b44a1b8f-kube-api-access-ptccw\" (UniqueName: \"kubernetes.io/projected/04b7431a-dbc2-47b6-92b3-da82b44a1b8f-kube-api-access-ptccw\") pod \"04b7431a-dbc2-47b6-92b3-da82b44a1b8f\" (UID: \"04b7431a-dbc2-47b6-92b3-da82b44a1b8f\") " Nov 01 20:33:56.713096 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:56.713071 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04b7431a-dbc2-47b6-92b3-da82b44a1b8f-plugins-config-volume" pod "04b7431a-dbc2-47b6-92b3-da82b44a1b8f" (UID: "04b7431a-dbc2-47b6-92b3-da82b44a1b8f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:33:56.714839 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:56.714817 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04b7431a-dbc2-47b6-92b3-da82b44a1b8f-kube-api-access-ptccw" pod "04b7431a-dbc2-47b6-92b3-da82b44a1b8f" (UID: "04b7431a-dbc2-47b6-92b3-da82b44a1b8f"). InnerVolumeSpecName "kube-api-access-ptccw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:33:56.814056 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:56.814018 5935 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04b7431a-dbc2-47b6-92b3-da82b44a1b8f-plugins-config-volume\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:33:56.814056 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:56.814050 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptccw\" (UniqueName: \"kubernetes.io/projected/04b7431a-dbc2-47b6-92b3-da82b44a1b8f-kube-api-access-ptccw\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:33:57.423036 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:57.422997 5935 generic.go:356] "Generic (PLEG): container finished" podID="04b7431a-dbc2-47b6-92b3-da82b44a1b8f" containerID="da1b3779c285509a75104127f4259c714290350f3c184984032c1f4b3285cc74" exitCode=0 Nov 01 20:33:57.423499 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:57.423046 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-5370bb-648c6878bc-j99qx" Nov 01 20:33:57.423499 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:57.423075 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-5370bb-648c6878bc-j99qx" event={"ID":"04b7431a-dbc2-47b6-92b3-da82b44a1b8f","Type":"ContainerDied","Data":"da1b3779c285509a75104127f4259c714290350f3c184984032c1f4b3285cc74"} Nov 01 20:33:57.423499 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:57.423127 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-5370bb-648c6878bc-j99qx" event={"ID":"04b7431a-dbc2-47b6-92b3-da82b44a1b8f","Type":"ContainerDied","Data":"dfd7ae0ff2ac07c3ca700b5f0c254f357aba60dbde22f7091b33d149c972dd74"} Nov 01 20:33:57.423499 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:57.423151 5935 scope.go:122] "RemoveContainer" containerID="da1b3779c285509a75104127f4259c714290350f3c184984032c1f4b3285cc74" Nov 01 20:33:57.432971 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:57.432946 5935 scope.go:122] "RemoveContainer" containerID="da1b3779c285509a75104127f4259c714290350f3c184984032c1f4b3285cc74" Nov 01 20:33:57.433224 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:33:57.433203 5935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da1b3779c285509a75104127f4259c714290350f3c184984032c1f4b3285cc74\": container with ID starting with da1b3779c285509a75104127f4259c714290350f3c184984032c1f4b3285cc74 not found: ID does not exist" containerID="da1b3779c285509a75104127f4259c714290350f3c184984032c1f4b3285cc74" Nov 01 20:33:57.433340 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:57.433235 5935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da1b3779c285509a75104127f4259c714290350f3c184984032c1f4b3285cc74"} err="failed to get container status \"da1b3779c285509a75104127f4259c714290350f3c184984032c1f4b3285cc74\": rpc error: code = NotFound desc = could not find container \"da1b3779c285509a75104127f4259c714290350f3c184984032c1f4b3285cc74\": container with ID starting with da1b3779c285509a75104127f4259c714290350f3c184984032c1f4b3285cc74 not found: ID does not exist" Nov 01 20:33:57.444099 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:57.444071 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-5370bb-648c6878bc-j99qx"] Nov 01 20:33:57.448009 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:57.447983 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-5370bb-648c6878bc-j99qx"] Nov 01 20:33:58.352608 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:33:58.352572 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04b7431a-dbc2-47b6-92b3-da82b44a1b8f" path="/var/lib/kubelet/pods/04b7431a-dbc2-47b6-92b3-da82b44a1b8f/volumes" Nov 01 20:34:01.763198 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:01.763167 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-x9p74_e4de1911-27fd-43cb-970b-308f8380bd6d/manager/0.log" Nov 01 20:34:02.147952 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:02.147853 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-vtpsz_458588d6-f7a6-4e19-a2e5-75bf2a9373fc/manager/0.log" Nov 01 20:34:02.770453 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:02.770411 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5370bb-6c5d87878c-kng8d" podUID="4b4d32cc-fa47-4b9d-9e8b-c39dfd020196" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 20:34:02.923822 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:02.923786 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-jbfsn_33b28b24-2fcc-40d9-9ad0-059f15acf4aa/server/0.log" Nov 01 20:34:03.296023 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:03.295991 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-9hj4l_557bbd53-195b-4a00-985e-d3f8647758d1/proxy/0.log" Nov 01 20:34:05.843723 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:34:05.843683 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:34:12.769908 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:12.769872 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5370bb-6c5d87878c-kng8d" podUID="4b4d32cc-fa47-4b9d-9e8b-c39dfd020196" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 20:34:16.525461 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:16.525417 5935 generic.go:356] "Generic (PLEG): container finished" podID="4b4d32cc-fa47-4b9d-9e8b-c39dfd020196" containerID="0d09088a310336b869185a4926d8b57aad71b987620f5e0eddf5d416c9ac2c90" exitCode=0 Nov 01 20:34:16.525852 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:16.525497 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5370bb-6c5d87878c-kng8d" event={"ID":"4b4d32cc-fa47-4b9d-9e8b-c39dfd020196","Type":"ContainerDied","Data":"0d09088a310336b869185a4926d8b57aad71b987620f5e0eddf5d416c9ac2c90"} Nov 01 20:34:16.640984 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:16.640958 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5370bb-6c5d87878c-kng8d" Nov 01 20:34:16.712529 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:16.712495 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b4d32cc-fa47-4b9d-9e8b-c39dfd020196-kube-api-access-2kpxx\" (UniqueName: \"kubernetes.io/projected/4b4d32cc-fa47-4b9d-9e8b-c39dfd020196-kube-api-access-2kpxx\") pod \"4b4d32cc-fa47-4b9d-9e8b-c39dfd020196\" (UID: \"4b4d32cc-fa47-4b9d-9e8b-c39dfd020196\") " Nov 01 20:34:16.712733 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:16.712552 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b4d32cc-fa47-4b9d-9e8b-c39dfd020196-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b4d32cc-fa47-4b9d-9e8b-c39dfd020196-plugins-config-volume\") pod \"4b4d32cc-fa47-4b9d-9e8b-c39dfd020196\" (UID: \"4b4d32cc-fa47-4b9d-9e8b-c39dfd020196\") " Nov 01 20:34:16.712990 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:16.712962 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b4d32cc-fa47-4b9d-9e8b-c39dfd020196-plugins-config-volume" pod "4b4d32cc-fa47-4b9d-9e8b-c39dfd020196" (UID: "4b4d32cc-fa47-4b9d-9e8b-c39dfd020196"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:34:16.714853 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:16.714830 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b4d32cc-fa47-4b9d-9e8b-c39dfd020196-kube-api-access-2kpxx" pod "4b4d32cc-fa47-4b9d-9e8b-c39dfd020196" (UID: "4b4d32cc-fa47-4b9d-9e8b-c39dfd020196"). InnerVolumeSpecName "kube-api-access-2kpxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:34:16.814049 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:16.814012 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kpxx\" (UniqueName: \"kubernetes.io/projected/4b4d32cc-fa47-4b9d-9e8b-c39dfd020196-kube-api-access-2kpxx\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:34:16.814049 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:16.814041 5935 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b4d32cc-fa47-4b9d-9e8b-c39dfd020196-plugins-config-volume\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:34:17.531550 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:17.531521 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5370bb-6c5d87878c-kng8d" Nov 01 20:34:17.531550 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:17.531552 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5370bb-6c5d87878c-kng8d" event={"ID":"4b4d32cc-fa47-4b9d-9e8b-c39dfd020196","Type":"ContainerDied","Data":"04b0c5b716534151ad4a57f092fad568e32d78a882d98490844f128fe0933868"} Nov 01 20:34:17.532130 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:17.531615 5935 scope.go:122] "RemoveContainer" containerID="0d09088a310336b869185a4926d8b57aad71b987620f5e0eddf5d416c9ac2c90" Nov 01 20:34:17.552598 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:17.552565 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-5370bb-6c5d87878c-kng8d"] Nov 01 20:34:17.555138 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:17.555108 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-5370bb-6c5d87878c-kng8d"] Nov 01 20:34:18.353793 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:18.353754 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b4d32cc-fa47-4b9d-9e8b-c39dfd020196" path="/var/lib/kubelet/pods/4b4d32cc-fa47-4b9d-9e8b-c39dfd020196/volumes" Nov 01 20:34:27.436775 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:27.436740 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-main-1bfa0e/simulator-main-1bfa0e-kserve-847dc858bb-k46l8"] Nov 01 20:34:27.437404 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:27.437336 5935 cpu_manager.go:414] "containerMap: removing container" podUID="e8bf3d0d-c668-44f1-9238-746e548faba0" containerName="istio-proxy" Nov 01 20:34:27.437404 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:27.437358 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8bf3d0d-c668-44f1-9238-746e548faba0" containerName="istio-proxy" Nov 01 20:34:27.437404 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:27.437377 5935 cpu_manager.go:414] "containerMap: removing container" podUID="81191553-1912-45b2-a58a-01fc64d43e23" containerName="payload-pre-processing" Nov 01 20:34:27.437404 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:27.437386 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81191553-1912-45b2-a58a-01fc64d43e23" containerName="payload-pre-processing" Nov 01 20:34:27.437404 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:27.437396 5935 cpu_manager.go:414] "containerMap: removing container" podUID="1238788e-4bf8-4a19-8de9-809061a7cb2b" containerName="maas-api" Nov 01 20:34:27.437404 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:27.437404 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1238788e-4bf8-4a19-8de9-809061a7cb2b" containerName="maas-api" Nov 01 20:34:27.437741 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:27.437417 5935 cpu_manager.go:414] "containerMap: removing container" podUID="04b7431a-dbc2-47b6-92b3-da82b44a1b8f" containerName="payload-processing" Nov 01 20:34:27.437741 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:27.437424 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04b7431a-dbc2-47b6-92b3-da82b44a1b8f" containerName="payload-processing" Nov 01 20:34:27.437741 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:27.437436 5935 cpu_manager.go:414] "containerMap: removing container" podUID="32159d48-5e0e-4841-a448-c57d21ab523f" containerName="maas-api" Nov 01 20:34:27.437741 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:27.437444 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32159d48-5e0e-4841-a448-c57d21ab523f" containerName="maas-api" Nov 01 20:34:27.437741 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:27.437459 5935 cpu_manager.go:414] "containerMap: removing container" podUID="7b4eb318-7389-46d3-a2b0-3f160182503f" containerName="istio-proxy" Nov 01 20:34:27.437741 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:27.437467 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b4eb318-7389-46d3-a2b0-3f160182503f" containerName="istio-proxy" Nov 01 20:34:27.437741 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:27.437477 5935 cpu_manager.go:414] "containerMap: removing container" podUID="4b4d32cc-fa47-4b9d-9e8b-c39dfd020196" containerName="payload-pre-processing" Nov 01 20:34:27.437741 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:27.437485 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b4d32cc-fa47-4b9d-9e8b-c39dfd020196" containerName="payload-pre-processing" Nov 01 20:34:27.437741 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:27.437609 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="04b7431a-dbc2-47b6-92b3-da82b44a1b8f" containerName="payload-processing" Nov 01 20:34:27.437741 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:27.437626 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="32159d48-5e0e-4841-a448-c57d21ab523f" containerName="maas-api" Nov 01 20:34:27.437741 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:27.437636 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b4eb318-7389-46d3-a2b0-3f160182503f" containerName="istio-proxy" Nov 01 20:34:27.437741 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:27.437646 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b4d32cc-fa47-4b9d-9e8b-c39dfd020196" containerName="payload-pre-processing" Nov 01 20:34:27.437741 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:27.437663 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8bf3d0d-c668-44f1-9238-746e548faba0" containerName="istio-proxy" Nov 01 20:34:27.437741 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:27.437673 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="81191553-1912-45b2-a58a-01fc64d43e23" containerName="payload-pre-processing" Nov 01 20:34:27.437741 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:27.437683 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="1238788e-4bf8-4a19-8de9-809061a7cb2b" containerName="maas-api" Nov 01 20:34:27.442084 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:27.442061 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-main-1bfa0e/simulator-main-1bfa0e-kserve-847dc858bb-k46l8" Nov 01 20:34:27.448592 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:27.448565 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-main-1bfa0e/simulator-main-1bfa0e-kserve-847dc858bb-k46l8"] Nov 01 20:34:27.449368 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:27.449345 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-main-1bfa0e\"/\"openshift-service-ca.crt\"" Nov 01 20:34:27.451249 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:27.451229 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-main-1bfa0e\"/\"default-dockercfg-7gxcq\"" Nov 01 20:34:27.451385 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:27.451303 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-main-1bfa0e\"/\"kube-root-ca.crt\"" Nov 01 20:34:27.451471 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:27.451437 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-main-1bfa0e\"/\"simulator-main-1bfa0e-kserve-self-signed-certs\"" Nov 01 20:34:27.517355 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:27.517321 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ksww\" (UniqueName: \"kubernetes.io/projected/5a9fbeb6-0f00-4a8c-9d76-f7550dd55775-kube-api-access-4ksww\") pod \"simulator-main-1bfa0e-kserve-847dc858bb-k46l8\" (UID: \"5a9fbeb6-0f00-4a8c-9d76-f7550dd55775\") " pod="e2e-models-e2e-worker-main-1bfa0e/simulator-main-1bfa0e-kserve-847dc858bb-k46l8" Nov 01 20:34:27.517534 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:27.517438 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a9fbeb6-0f00-4a8c-9d76-f7550dd55775-dshm\") pod \"simulator-main-1bfa0e-kserve-847dc858bb-k46l8\" (UID: \"5a9fbeb6-0f00-4a8c-9d76-f7550dd55775\") " pod="e2e-models-e2e-worker-main-1bfa0e/simulator-main-1bfa0e-kserve-847dc858bb-k46l8" Nov 01 20:34:27.517589 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:27.517530 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a9fbeb6-0f00-4a8c-9d76-f7550dd55775-tmp-dir\") pod \"simulator-main-1bfa0e-kserve-847dc858bb-k46l8\" (UID: \"5a9fbeb6-0f00-4a8c-9d76-f7550dd55775\") " pod="e2e-models-e2e-worker-main-1bfa0e/simulator-main-1bfa0e-kserve-847dc858bb-k46l8" Nov 01 20:34:27.517649 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:27.517633 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a9fbeb6-0f00-4a8c-9d76-f7550dd55775-home\") pod \"simulator-main-1bfa0e-kserve-847dc858bb-k46l8\" (UID: \"5a9fbeb6-0f00-4a8c-9d76-f7550dd55775\") " pod="e2e-models-e2e-worker-main-1bfa0e/simulator-main-1bfa0e-kserve-847dc858bb-k46l8" Nov 01 20:34:27.517754 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:27.517729 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a9fbeb6-0f00-4a8c-9d76-f7550dd55775-tls-certs\") pod \"simulator-main-1bfa0e-kserve-847dc858bb-k46l8\" (UID: \"5a9fbeb6-0f00-4a8c-9d76-f7550dd55775\") " pod="e2e-models-e2e-worker-main-1bfa0e/simulator-main-1bfa0e-kserve-847dc858bb-k46l8" Nov 01 20:34:27.517796 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:27.517774 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a9fbeb6-0f00-4a8c-9d76-f7550dd55775-model-cache\") pod \"simulator-main-1bfa0e-kserve-847dc858bb-k46l8\" (UID: \"5a9fbeb6-0f00-4a8c-9d76-f7550dd55775\") " pod="e2e-models-e2e-worker-main-1bfa0e/simulator-main-1bfa0e-kserve-847dc858bb-k46l8" Nov 01 20:34:27.619534 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:27.619486 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ksww\" (UniqueName: \"kubernetes.io/projected/5a9fbeb6-0f00-4a8c-9d76-f7550dd55775-kube-api-access-4ksww\") pod \"simulator-main-1bfa0e-kserve-847dc858bb-k46l8\" (UID: \"5a9fbeb6-0f00-4a8c-9d76-f7550dd55775\") " pod="e2e-models-e2e-worker-main-1bfa0e/simulator-main-1bfa0e-kserve-847dc858bb-k46l8" Nov 01 20:34:27.619738 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:27.619583 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a9fbeb6-0f00-4a8c-9d76-f7550dd55775-dshm\") pod \"simulator-main-1bfa0e-kserve-847dc858bb-k46l8\" (UID: \"5a9fbeb6-0f00-4a8c-9d76-f7550dd55775\") " pod="e2e-models-e2e-worker-main-1bfa0e/simulator-main-1bfa0e-kserve-847dc858bb-k46l8" Nov 01 20:34:27.619738 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:27.619613 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a9fbeb6-0f00-4a8c-9d76-f7550dd55775-tmp-dir\") pod \"simulator-main-1bfa0e-kserve-847dc858bb-k46l8\" (UID: \"5a9fbeb6-0f00-4a8c-9d76-f7550dd55775\") " pod="e2e-models-e2e-worker-main-1bfa0e/simulator-main-1bfa0e-kserve-847dc858bb-k46l8" Nov 01 20:34:27.619738 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:27.619666 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a9fbeb6-0f00-4a8c-9d76-f7550dd55775-home\") pod \"simulator-main-1bfa0e-kserve-847dc858bb-k46l8\" (UID: \"5a9fbeb6-0f00-4a8c-9d76-f7550dd55775\") " pod="e2e-models-e2e-worker-main-1bfa0e/simulator-main-1bfa0e-kserve-847dc858bb-k46l8" Nov 01 20:34:27.619738 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:27.619711 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a9fbeb6-0f00-4a8c-9d76-f7550dd55775-tls-certs\") pod \"simulator-main-1bfa0e-kserve-847dc858bb-k46l8\" (UID: \"5a9fbeb6-0f00-4a8c-9d76-f7550dd55775\") " pod="e2e-models-e2e-worker-main-1bfa0e/simulator-main-1bfa0e-kserve-847dc858bb-k46l8" Nov 01 20:34:27.619738 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:27.619730 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a9fbeb6-0f00-4a8c-9d76-f7550dd55775-model-cache\") pod \"simulator-main-1bfa0e-kserve-847dc858bb-k46l8\" (UID: \"5a9fbeb6-0f00-4a8c-9d76-f7550dd55775\") " pod="e2e-models-e2e-worker-main-1bfa0e/simulator-main-1bfa0e-kserve-847dc858bb-k46l8" Nov 01 20:34:27.620131 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:27.620107 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a9fbeb6-0f00-4a8c-9d76-f7550dd55775-model-cache\") pod \"simulator-main-1bfa0e-kserve-847dc858bb-k46l8\" (UID: \"5a9fbeb6-0f00-4a8c-9d76-f7550dd55775\") " pod="e2e-models-e2e-worker-main-1bfa0e/simulator-main-1bfa0e-kserve-847dc858bb-k46l8" Nov 01 20:34:27.620955 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:27.620931 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a9fbeb6-0f00-4a8c-9d76-f7550dd55775-home\") pod \"simulator-main-1bfa0e-kserve-847dc858bb-k46l8\" (UID: \"5a9fbeb6-0f00-4a8c-9d76-f7550dd55775\") " pod="e2e-models-e2e-worker-main-1bfa0e/simulator-main-1bfa0e-kserve-847dc858bb-k46l8" Nov 01 20:34:27.621110 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:27.620980 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a9fbeb6-0f00-4a8c-9d76-f7550dd55775-tmp-dir\") pod \"simulator-main-1bfa0e-kserve-847dc858bb-k46l8\" (UID: \"5a9fbeb6-0f00-4a8c-9d76-f7550dd55775\") " pod="e2e-models-e2e-worker-main-1bfa0e/simulator-main-1bfa0e-kserve-847dc858bb-k46l8" Nov 01 20:34:27.623140 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:27.623118 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a9fbeb6-0f00-4a8c-9d76-f7550dd55775-dshm\") pod \"simulator-main-1bfa0e-kserve-847dc858bb-k46l8\" (UID: \"5a9fbeb6-0f00-4a8c-9d76-f7550dd55775\") " pod="e2e-models-e2e-worker-main-1bfa0e/simulator-main-1bfa0e-kserve-847dc858bb-k46l8" Nov 01 20:34:27.623414 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:27.623393 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a9fbeb6-0f00-4a8c-9d76-f7550dd55775-tls-certs\") pod \"simulator-main-1bfa0e-kserve-847dc858bb-k46l8\" (UID: \"5a9fbeb6-0f00-4a8c-9d76-f7550dd55775\") " pod="e2e-models-e2e-worker-main-1bfa0e/simulator-main-1bfa0e-kserve-847dc858bb-k46l8" Nov 01 20:34:27.628354 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:27.628333 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ksww\" (UniqueName: \"kubernetes.io/projected/5a9fbeb6-0f00-4a8c-9d76-f7550dd55775-kube-api-access-4ksww\") pod \"simulator-main-1bfa0e-kserve-847dc858bb-k46l8\" (UID: \"5a9fbeb6-0f00-4a8c-9d76-f7550dd55775\") " pod="e2e-models-e2e-worker-main-1bfa0e/simulator-main-1bfa0e-kserve-847dc858bb-k46l8" Nov 01 20:34:27.756955 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:27.756925 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-main-1bfa0e/simulator-main-1bfa0e-kserve-847dc858bb-k46l8" Nov 01 20:34:28.033115 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:28.033090 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-main-1bfa0e/simulator-main-1bfa0e-kserve-847dc858bb-k46l8"] Nov 01 20:34:28.035990 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:34:28.035965 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a9fbeb6_0f00_4a8c_9d76_f7550dd55775.slice/crio-972afc34b75308d90951edba6b0f8853e381b6429ab641aae9e7ffcffe270d0b WatchSource:0}: Error finding container 972afc34b75308d90951edba6b0f8853e381b6429ab641aae9e7ffcffe270d0b: Status 404 returned error can't find the container with id 972afc34b75308d90951edba6b0f8853e381b6429ab641aae9e7ffcffe270d0b Nov 01 20:34:28.590161 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:28.590122 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-main-1bfa0e/simulator-main-1bfa0e-kserve-847dc858bb-k46l8" event={"ID":"5a9fbeb6-0f00-4a8c-9d76-f7550dd55775","Type":"ContainerStarted","Data":"0cba2a7b9e9c49e590a6b60aa58f9f4d1c4d9020bb003da8c474bba8c49e5f37"} Nov 01 20:34:28.590161 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:28.590172 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-main-1bfa0e/simulator-main-1bfa0e-kserve-847dc858bb-k46l8" event={"ID":"5a9fbeb6-0f00-4a8c-9d76-f7550dd55775","Type":"ContainerStarted","Data":"972afc34b75308d90951edba6b0f8853e381b6429ab641aae9e7ffcffe270d0b"} Nov 01 20:34:28.606342 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:28.606301 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-main-1bfa0e/simulator-main-1bfa0e-kserve-847dc858bb-k46l8" podStartSLOduration=1.60627401 podStartE2EDuration="1.60627401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:34:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:34:28.605066286 +0000 UTC m=+2114.378226811" watchObservedRunningTime="2026-11-01 20:34:28.60627401 +0000 UTC m=+2114.379434521" Nov 01 20:34:28.757757 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:28.757721 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-main-1bfa0e/simulator-main-1bfa0e-kserve-847dc858bb-k46l8" Nov 01 20:34:35.843607 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:34:35.843570 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:34:37.757524 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:37.757494 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-main-1bfa0e/simulator-main-1bfa0e-kserve-847dc858bb-k46l8" Nov 01 20:34:37.769794 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:37.769773 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-main-1bfa0e/simulator-main-1bfa0e-kserve-847dc858bb-k46l8" Nov 01 20:34:38.656651 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:38.656621 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-main-1bfa0e/simulator-main-1bfa0e-kserve-847dc858bb-k46l8" Nov 01 20:34:44.021571 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:44.021536 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-main-1bfa0e/premium-main-1bfa0e-kserve-546887c779-lbfjg"] Nov 01 20:34:44.025701 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:44.025679 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-main-1bfa0e/premium-main-1bfa0e-kserve-546887c779-lbfjg" Nov 01 20:34:44.033409 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:44.033381 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-main-1bfa0e\"/\"premium-main-1bfa0e-kserve-self-signed-certs\"" Nov 01 20:34:44.035831 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:44.035803 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-main-1bfa0e/premium-main-1bfa0e-kserve-546887c779-lbfjg"] Nov 01 20:34:44.172542 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:44.172507 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ac5ece4-dabf-4248-892e-c139c3103ac9-home\") pod \"premium-main-1bfa0e-kserve-546887c779-lbfjg\" (UID: \"5ac5ece4-dabf-4248-892e-c139c3103ac9\") " pod="e2e-models-e2e-worker-main-1bfa0e/premium-main-1bfa0e-kserve-546887c779-lbfjg" Nov 01 20:34:44.172748 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:44.172642 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ac5ece4-dabf-4248-892e-c139c3103ac9-tls-certs\") pod \"premium-main-1bfa0e-kserve-546887c779-lbfjg\" (UID: \"5ac5ece4-dabf-4248-892e-c139c3103ac9\") " pod="e2e-models-e2e-worker-main-1bfa0e/premium-main-1bfa0e-kserve-546887c779-lbfjg" Nov 01 20:34:44.172748 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:44.172685 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ac5ece4-dabf-4248-892e-c139c3103ac9-dshm\") pod \"premium-main-1bfa0e-kserve-546887c779-lbfjg\" (UID: \"5ac5ece4-dabf-4248-892e-c139c3103ac9\") " pod="e2e-models-e2e-worker-main-1bfa0e/premium-main-1bfa0e-kserve-546887c779-lbfjg" Nov 01 20:34:44.172748 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:44.172742 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ac5ece4-dabf-4248-892e-c139c3103ac9-tmp-dir\") pod \"premium-main-1bfa0e-kserve-546887c779-lbfjg\" (UID: \"5ac5ece4-dabf-4248-892e-c139c3103ac9\") " pod="e2e-models-e2e-worker-main-1bfa0e/premium-main-1bfa0e-kserve-546887c779-lbfjg" Nov 01 20:34:44.172921 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:44.172775 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfk2b\" (UniqueName: \"kubernetes.io/projected/5ac5ece4-dabf-4248-892e-c139c3103ac9-kube-api-access-gfk2b\") pod \"premium-main-1bfa0e-kserve-546887c779-lbfjg\" (UID: \"5ac5ece4-dabf-4248-892e-c139c3103ac9\") " pod="e2e-models-e2e-worker-main-1bfa0e/premium-main-1bfa0e-kserve-546887c779-lbfjg" Nov 01 20:34:44.172921 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:44.172817 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ac5ece4-dabf-4248-892e-c139c3103ac9-model-cache\") pod \"premium-main-1bfa0e-kserve-546887c779-lbfjg\" (UID: \"5ac5ece4-dabf-4248-892e-c139c3103ac9\") " pod="e2e-models-e2e-worker-main-1bfa0e/premium-main-1bfa0e-kserve-546887c779-lbfjg" Nov 01 20:34:44.273447 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:44.273371 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ac5ece4-dabf-4248-892e-c139c3103ac9-tls-certs\") pod \"premium-main-1bfa0e-kserve-546887c779-lbfjg\" (UID: \"5ac5ece4-dabf-4248-892e-c139c3103ac9\") " pod="e2e-models-e2e-worker-main-1bfa0e/premium-main-1bfa0e-kserve-546887c779-lbfjg" Nov 01 20:34:44.273447 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:44.273417 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ac5ece4-dabf-4248-892e-c139c3103ac9-dshm\") pod \"premium-main-1bfa0e-kserve-546887c779-lbfjg\" (UID: \"5ac5ece4-dabf-4248-892e-c139c3103ac9\") " pod="e2e-models-e2e-worker-main-1bfa0e/premium-main-1bfa0e-kserve-546887c779-lbfjg" Nov 01 20:34:44.273447 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:44.273437 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ac5ece4-dabf-4248-892e-c139c3103ac9-tmp-dir\") pod \"premium-main-1bfa0e-kserve-546887c779-lbfjg\" (UID: \"5ac5ece4-dabf-4248-892e-c139c3103ac9\") " pod="e2e-models-e2e-worker-main-1bfa0e/premium-main-1bfa0e-kserve-546887c779-lbfjg" Nov 01 20:34:44.273714 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:44.273455 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfk2b\" (UniqueName: \"kubernetes.io/projected/5ac5ece4-dabf-4248-892e-c139c3103ac9-kube-api-access-gfk2b\") pod \"premium-main-1bfa0e-kserve-546887c779-lbfjg\" (UID: \"5ac5ece4-dabf-4248-892e-c139c3103ac9\") " pod="e2e-models-e2e-worker-main-1bfa0e/premium-main-1bfa0e-kserve-546887c779-lbfjg" Nov 01 20:34:44.273714 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:44.273489 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ac5ece4-dabf-4248-892e-c139c3103ac9-model-cache\") pod \"premium-main-1bfa0e-kserve-546887c779-lbfjg\" (UID: \"5ac5ece4-dabf-4248-892e-c139c3103ac9\") " pod="e2e-models-e2e-worker-main-1bfa0e/premium-main-1bfa0e-kserve-546887c779-lbfjg" Nov 01 20:34:44.273714 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:44.273514 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ac5ece4-dabf-4248-892e-c139c3103ac9-home\") pod \"premium-main-1bfa0e-kserve-546887c779-lbfjg\" (UID: \"5ac5ece4-dabf-4248-892e-c139c3103ac9\") " pod="e2e-models-e2e-worker-main-1bfa0e/premium-main-1bfa0e-kserve-546887c779-lbfjg" Nov 01 20:34:44.273915 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:44.273847 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ac5ece4-dabf-4248-892e-c139c3103ac9-tmp-dir\") pod \"premium-main-1bfa0e-kserve-546887c779-lbfjg\" (UID: \"5ac5ece4-dabf-4248-892e-c139c3103ac9\") " pod="e2e-models-e2e-worker-main-1bfa0e/premium-main-1bfa0e-kserve-546887c779-lbfjg" Nov 01 20:34:44.273984 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:44.273960 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ac5ece4-dabf-4248-892e-c139c3103ac9-model-cache\") pod \"premium-main-1bfa0e-kserve-546887c779-lbfjg\" (UID: \"5ac5ece4-dabf-4248-892e-c139c3103ac9\") " pod="e2e-models-e2e-worker-main-1bfa0e/premium-main-1bfa0e-kserve-546887c779-lbfjg" Nov 01 20:34:44.274062 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:44.274046 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ac5ece4-dabf-4248-892e-c139c3103ac9-home\") pod \"premium-main-1bfa0e-kserve-546887c779-lbfjg\" (UID: \"5ac5ece4-dabf-4248-892e-c139c3103ac9\") " pod="e2e-models-e2e-worker-main-1bfa0e/premium-main-1bfa0e-kserve-546887c779-lbfjg" Nov 01 20:34:44.275902 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:44.275883 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ac5ece4-dabf-4248-892e-c139c3103ac9-tls-certs\") pod \"premium-main-1bfa0e-kserve-546887c779-lbfjg\" (UID: \"5ac5ece4-dabf-4248-892e-c139c3103ac9\") " pod="e2e-models-e2e-worker-main-1bfa0e/premium-main-1bfa0e-kserve-546887c779-lbfjg" Nov 01 20:34:44.276031 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:44.276012 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ac5ece4-dabf-4248-892e-c139c3103ac9-dshm\") pod \"premium-main-1bfa0e-kserve-546887c779-lbfjg\" (UID: \"5ac5ece4-dabf-4248-892e-c139c3103ac9\") " pod="e2e-models-e2e-worker-main-1bfa0e/premium-main-1bfa0e-kserve-546887c779-lbfjg" Nov 01 20:34:44.281201 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:44.281178 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfk2b\" (UniqueName: \"kubernetes.io/projected/5ac5ece4-dabf-4248-892e-c139c3103ac9-kube-api-access-gfk2b\") pod \"premium-main-1bfa0e-kserve-546887c779-lbfjg\" (UID: \"5ac5ece4-dabf-4248-892e-c139c3103ac9\") " pod="e2e-models-e2e-worker-main-1bfa0e/premium-main-1bfa0e-kserve-546887c779-lbfjg" Nov 01 20:34:44.338493 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:44.338469 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-main-1bfa0e/premium-main-1bfa0e-kserve-546887c779-lbfjg" Nov 01 20:34:44.614048 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:44.613979 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-main-1bfa0e/premium-main-1bfa0e-kserve-546887c779-lbfjg"] Nov 01 20:34:44.618583 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:34:44.618545 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5ac5ece4_dabf_4248_892e_c139c3103ac9.slice/crio-531c01558d6586e7b88a27cfeacbf5dd033f8dca0460f81d208842e18f44f41c WatchSource:0}: Error finding container 531c01558d6586e7b88a27cfeacbf5dd033f8dca0460f81d208842e18f44f41c: Status 404 returned error can't find the container with id 531c01558d6586e7b88a27cfeacbf5dd033f8dca0460f81d208842e18f44f41c Nov 01 20:34:44.683325 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:44.683299 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-main-1bfa0e/premium-main-1bfa0e-kserve-546887c779-lbfjg" event={"ID":"5ac5ece4-dabf-4248-892e-c139c3103ac9","Type":"ContainerStarted","Data":"72316667f4213f1440f02583d8d28880e3e97814c09fc80da31de6d7804be4e4"} Nov 01 20:34:44.683425 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:44.683341 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-main-1bfa0e/premium-main-1bfa0e-kserve-546887c779-lbfjg" event={"ID":"5ac5ece4-dabf-4248-892e-c139c3103ac9","Type":"ContainerStarted","Data":"531c01558d6586e7b88a27cfeacbf5dd033f8dca0460f81d208842e18f44f41c"} Nov 01 20:34:44.699222 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:44.699168 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-main-1bfa0e/premium-main-1bfa0e-kserve-546887c779-lbfjg" podStartSLOduration=0.699151858 podStartE2EDuration="699.151858ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:34:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:34:44.697903155 +0000 UTC m=+2130.471063668" watchObservedRunningTime="2026-11-01 20:34:44.699151858 +0000 UTC m=+2130.472312371" Nov 01 20:34:45.339492 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:45.339445 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-main-1bfa0e/premium-main-1bfa0e-kserve-546887c779-lbfjg" Nov 01 20:34:54.339148 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:54.339115 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-main-1bfa0e/premium-main-1bfa0e-kserve-546887c779-lbfjg" Nov 01 20:34:54.352921 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:54.352899 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-main-1bfa0e/premium-main-1bfa0e-kserve-546887c779-lbfjg" Nov 01 20:34:54.747213 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:34:54.747188 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-main-1bfa0e/premium-main-1bfa0e-kserve-546887c779-lbfjg" Nov 01 20:35:05.843243 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:35:05.843207 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:35:35.842664 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:35:35.842630 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:36:05.842598 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:36:05.842562 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:36:35.845948 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:36:35.845914 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:37:05.843632 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:37:05.843598 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:37:35.843109 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:37:35.843074 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:37:50.017183 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:37:50.017148 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-fl2rj"] Nov 01 20:37:50.020567 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:37:50.020550 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-fl2rj" Nov 01 20:37:50.025236 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:37:50.025213 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:37:50.026434 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:37:50.026414 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tq9vg\"" Nov 01 20:37:50.028637 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:37:50.028619 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-fl2rj"] Nov 01 20:37:50.030846 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:37:50.030825 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38a5cb76-70ed-4e9e-b6c8-a4e794f500cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-fl2rj\" (UID: \"38a5cb76-70ed-4e9e-b6c8-a4e794f500cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-fl2rj" Nov 01 20:37:50.030943 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:37:50.030864 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmtfb\" (UniqueName: \"kubernetes.io/projected/38a5cb76-70ed-4e9e-b6c8-a4e794f500cc-kube-api-access-pmtfb\") pod \"maas-api-revoke-sub-d576e68b-fl2rj\" (UID: \"38a5cb76-70ed-4e9e-b6c8-a4e794f500cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-fl2rj" Nov 01 20:37:50.030943 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:37:50.030892 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38a5cb76-70ed-4e9e-b6c8-a4e794f500cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-fl2rj\" (UID: \"38a5cb76-70ed-4e9e-b6c8-a4e794f500cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-fl2rj" Nov 01 20:37:50.131753 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:37:50.131727 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38a5cb76-70ed-4e9e-b6c8-a4e794f500cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-fl2rj\" (UID: \"38a5cb76-70ed-4e9e-b6c8-a4e794f500cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-fl2rj" Nov 01 20:37:50.131886 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:37:50.131769 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmtfb\" (UniqueName: \"kubernetes.io/projected/38a5cb76-70ed-4e9e-b6c8-a4e794f500cc-kube-api-access-pmtfb\") pod \"maas-api-revoke-sub-d576e68b-fl2rj\" (UID: \"38a5cb76-70ed-4e9e-b6c8-a4e794f500cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-fl2rj" Nov 01 20:37:50.131886 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:37:50.131793 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38a5cb76-70ed-4e9e-b6c8-a4e794f500cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-fl2rj\" (UID: \"38a5cb76-70ed-4e9e-b6c8-a4e794f500cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-fl2rj" Nov 01 20:37:50.132428 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:37:50.132404 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38a5cb76-70ed-4e9e-b6c8-a4e794f500cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-fl2rj\" (UID: \"38a5cb76-70ed-4e9e-b6c8-a4e794f500cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-fl2rj" Nov 01 20:37:50.134096 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:37:50.134077 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38a5cb76-70ed-4e9e-b6c8-a4e794f500cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-fl2rj\" (UID: \"38a5cb76-70ed-4e9e-b6c8-a4e794f500cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-fl2rj" Nov 01 20:37:50.142001 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:37:50.141982 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmtfb\" (UniqueName: \"kubernetes.io/projected/38a5cb76-70ed-4e9e-b6c8-a4e794f500cc-kube-api-access-pmtfb\") pod \"maas-api-revoke-sub-d576e68b-fl2rj\" (UID: \"38a5cb76-70ed-4e9e-b6c8-a4e794f500cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-fl2rj" Nov 01 20:37:50.333915 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:37:50.333828 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-fl2rj" Nov 01 20:37:50.608684 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:37:50.608659 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-fl2rj"] Nov 01 20:37:50.611387 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:37:50.611355 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38a5cb76_70ed_4e9e_b6c8_a4e794f500cc.slice/crio-8b5932194ce988fd6b391312da9f9982fdb717ad1da4fab2e03d061b3da579af WatchSource:0}: Error finding container 8b5932194ce988fd6b391312da9f9982fdb717ad1da4fab2e03d061b3da579af: Status 404 returned error can't find the container with id 8b5932194ce988fd6b391312da9f9982fdb717ad1da4fab2e03d061b3da579af Nov 01 20:37:50.628224 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:37:50.628187 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-fl2rj" event={"ID":"38a5cb76-70ed-4e9e-b6c8-a4e794f500cc","Type":"ContainerStarted","Data":"8b5932194ce988fd6b391312da9f9982fdb717ad1da4fab2e03d061b3da579af"} Nov 01 20:37:51.634697 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:37:51.634615 5935 generic.go:356] "Generic (PLEG): container finished" podID="38a5cb76-70ed-4e9e-b6c8-a4e794f500cc" containerID="84b94ba4bd0478a31b2b6fa8cbcb0c9d10b1d1bb07ad344e55b22bffab53f0e4" exitCode=0 Nov 01 20:37:51.635059 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:37:51.634703 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-fl2rj" event={"ID":"38a5cb76-70ed-4e9e-b6c8-a4e794f500cc","Type":"ContainerDied","Data":"84b94ba4bd0478a31b2b6fa8cbcb0c9d10b1d1bb07ad344e55b22bffab53f0e4"} Nov 01 20:37:52.768922 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:37:52.768894 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-fl2rj" Nov 01 20:37:52.852821 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:37:52.852782 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38a5cb76-70ed-4e9e-b6c8-a4e794f500cc-kube-api-access-pmtfb\" (UniqueName: \"kubernetes.io/projected/38a5cb76-70ed-4e9e-b6c8-a4e794f500cc-kube-api-access-pmtfb\") pod \"38a5cb76-70ed-4e9e-b6c8-a4e794f500cc\" (UID: \"38a5cb76-70ed-4e9e-b6c8-a4e794f500cc\") " Nov 01 20:37:52.853020 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:37:52.852897 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38a5cb76-70ed-4e9e-b6c8-a4e794f500cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38a5cb76-70ed-4e9e-b6c8-a4e794f500cc-maas-api-xks-ca\") pod \"38a5cb76-70ed-4e9e-b6c8-a4e794f500cc\" (UID: \"38a5cb76-70ed-4e9e-b6c8-a4e794f500cc\") " Nov 01 20:37:52.853020 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:37:52.852997 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38a5cb76-70ed-4e9e-b6c8-a4e794f500cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38a5cb76-70ed-4e9e-b6c8-a4e794f500cc-maas-api-openshift-ca\") pod \"38a5cb76-70ed-4e9e-b6c8-a4e794f500cc\" (UID: \"38a5cb76-70ed-4e9e-b6c8-a4e794f500cc\") " Nov 01 20:37:52.853368 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:37:52.853341 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38a5cb76-70ed-4e9e-b6c8-a4e794f500cc-maas-api-openshift-ca" pod "38a5cb76-70ed-4e9e-b6c8-a4e794f500cc" (UID: "38a5cb76-70ed-4e9e-b6c8-a4e794f500cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:37:52.854889 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:37:52.854867 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38a5cb76-70ed-4e9e-b6c8-a4e794f500cc-kube-api-access-pmtfb" pod "38a5cb76-70ed-4e9e-b6c8-a4e794f500cc" (UID: "38a5cb76-70ed-4e9e-b6c8-a4e794f500cc"). InnerVolumeSpecName "kube-api-access-pmtfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:37:52.854976 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:37:52.854953 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38a5cb76-70ed-4e9e-b6c8-a4e794f500cc-maas-api-xks-ca" pod "38a5cb76-70ed-4e9e-b6c8-a4e794f500cc" (UID: "38a5cb76-70ed-4e9e-b6c8-a4e794f500cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:37:52.953618 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:37:52.953592 5935 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38a5cb76-70ed-4e9e-b6c8-a4e794f500cc-maas-api-openshift-ca\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:37:52.953618 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:37:52.953617 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmtfb\" (UniqueName: \"kubernetes.io/projected/38a5cb76-70ed-4e9e-b6c8-a4e794f500cc-kube-api-access-pmtfb\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:37:52.953780 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:37:52.953627 5935 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38a5cb76-70ed-4e9e-b6c8-a4e794f500cc-maas-api-xks-ca\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:37:53.646319 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:37:53.646271 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-fl2rj" Nov 01 20:37:53.646319 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:37:53.646281 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-fl2rj" event={"ID":"38a5cb76-70ed-4e9e-b6c8-a4e794f500cc","Type":"ContainerDied","Data":"8b5932194ce988fd6b391312da9f9982fdb717ad1da4fab2e03d061b3da579af"} Nov 01 20:37:53.646546 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:37:53.646347 5935 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b5932194ce988fd6b391312da9f9982fdb717ad1da4fab2e03d061b3da579af" Nov 01 20:38:05.889910 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:38:05.889862 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:38:35.846750 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:38:35.846649 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:39:05.845100 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:39:05.845058 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:39:17.610310 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:39:17.608378 5935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-m6qv8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 20:39:17.610310 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:39:17.608445 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6qv8" podUID="9de91381-a29d-422d-b859-69229e8970c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:39:17.804392 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:39:17.804357 5935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-fl2rn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 20:39:17.804683 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:39:17.804653 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-fl2rn" podUID="043fcc04-22eb-454f-a1d5-d669214c5b0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:39:35.848188 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:39:35.848147 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:40:05.843729 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:40:05.843693 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:40:32.623156 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:40:32.623118 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-main-1bfa0e/premium-main-1bfa0e-kserve-546887c779-lbfjg"] Nov 01 20:40:32.627714 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:40:32.627684 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-main-1bfa0e/simulator-main-1bfa0e-kserve-847dc858bb-k46l8"] Nov 01 20:40:35.842959 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:40:35.842924 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:40:47.644978 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:40:47.644923 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-main-1bfa0e/premium-main-1bfa0e-kserve-546887c779-lbfjg" podUID="5ac5ece4-dabf-4248-892e-c139c3103ac9" containerName="main" containerID="cri-o://72316667f4213f1440f02583d8d28880e3e97814c09fc80da31de6d7804be4e4" gracePeriod=45 Nov 01 20:40:47.647677 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:40:47.647563 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-main-1bfa0e/simulator-main-1bfa0e-kserve-847dc858bb-k46l8" podUID="5a9fbeb6-0f00-4a8c-9d76-f7550dd55775" containerName="main" containerID="cri-o://0cba2a7b9e9c49e590a6b60aa58f9f4d1c4d9020bb003da8c474bba8c49e5f37" gracePeriod=45 Nov 01 20:40:47.652216 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:40:47.652185 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-main-1bfa0e/simulator-main-1bfa0e-kserve-847dc858bb-k46l8" podUID="5a9fbeb6-0f00-4a8c-9d76-f7550dd55775" containerName="main" probeResult="failure" output="Get \"https://10.133.0.118:8000/ready\": EOF" Nov 01 20:40:47.737208 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:40:47.737178 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-main-1bfa0e/premium-main-1bfa0e-kserve-546887c779-lbfjg" podUID="5ac5ece4-dabf-4248-892e-c139c3103ac9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.119:8000/ready\": dial tcp 10.133.0.119:8000: connect: connection refused" Nov 01 20:40:47.893200 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:40:47.893172 5935 generic.go:356] "Generic (PLEG): container finished" podID="5ac5ece4-dabf-4248-892e-c139c3103ac9" containerID="72316667f4213f1440f02583d8d28880e3e97814c09fc80da31de6d7804be4e4" exitCode=0 Nov 01 20:40:47.893338 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:40:47.893244 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-main-1bfa0e/premium-main-1bfa0e-kserve-546887c779-lbfjg" event={"ID":"5ac5ece4-dabf-4248-892e-c139c3103ac9","Type":"ContainerDied","Data":"72316667f4213f1440f02583d8d28880e3e97814c09fc80da31de6d7804be4e4"} Nov 01 20:40:47.893338 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:40:47.893305 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-main-1bfa0e/premium-main-1bfa0e-kserve-546887c779-lbfjg" event={"ID":"5ac5ece4-dabf-4248-892e-c139c3103ac9","Type":"ContainerDied","Data":"531c01558d6586e7b88a27cfeacbf5dd033f8dca0460f81d208842e18f44f41c"} Nov 01 20:40:47.893338 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:40:47.893323 5935 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="531c01558d6586e7b88a27cfeacbf5dd033f8dca0460f81d208842e18f44f41c" Nov 01 20:40:47.894672 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:40:47.894653 5935 generic.go:356] "Generic (PLEG): container finished" podID="5a9fbeb6-0f00-4a8c-9d76-f7550dd55775" containerID="0cba2a7b9e9c49e590a6b60aa58f9f4d1c4d9020bb003da8c474bba8c49e5f37" exitCode=0 Nov 01 20:40:47.894768 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:40:47.894682 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-main-1bfa0e/simulator-main-1bfa0e-kserve-847dc858bb-k46l8" event={"ID":"5a9fbeb6-0f00-4a8c-9d76-f7550dd55775","Type":"ContainerDied","Data":"0cba2a7b9e9c49e590a6b60aa58f9f4d1c4d9020bb003da8c474bba8c49e5f37"} Nov 01 20:40:47.911101 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:40:47.911078 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-main-1bfa0e/premium-main-1bfa0e-kserve-546887c779-lbfjg" Nov 01 20:40:47.914501 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:40:47.914478 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-main-1bfa0e/simulator-main-1bfa0e-kserve-847dc858bb-k46l8" Nov 01 20:40:48.024345 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:40:48.024310 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a9fbeb6-0f00-4a8c-9d76-f7550dd55775-home\" (UniqueName: \"kubernetes.io/empty-dir/5a9fbeb6-0f00-4a8c-9d76-f7550dd55775-home\") pod \"5a9fbeb6-0f00-4a8c-9d76-f7550dd55775\" (UID: \"5a9fbeb6-0f00-4a8c-9d76-f7550dd55775\") " Nov 01 20:40:48.024520 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:40:48.024366 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ac5ece4-dabf-4248-892e-c139c3103ac9-tls-certs\" (UniqueName: \"kubernetes.io/secret/5ac5ece4-dabf-4248-892e-c139c3103ac9-tls-certs\") pod \"5ac5ece4-dabf-4248-892e-c139c3103ac9\" (UID: \"5ac5ece4-dabf-4248-892e-c139c3103ac9\") " Nov 01 20:40:48.024520 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:40:48.024402 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ac5ece4-dabf-4248-892e-c139c3103ac9-home\" (UniqueName: \"kubernetes.io/empty-dir/5ac5ece4-dabf-4248-892e-c139c3103ac9-home\") pod \"5ac5ece4-dabf-4248-892e-c139c3103ac9\" (UID: \"5ac5ece4-dabf-4248-892e-c139c3103ac9\") " Nov 01 20:40:48.024520 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:40:48.024461 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a9fbeb6-0f00-4a8c-9d76-f7550dd55775-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a9fbeb6-0f00-4a8c-9d76-f7550dd55775-model-cache\") pod \"5a9fbeb6-0f00-4a8c-9d76-f7550dd55775\" (UID: \"5a9fbeb6-0f00-4a8c-9d76-f7550dd55775\") " Nov 01 20:40:48.024701 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:40:48.024570 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ac5ece4-dabf-4248-892e-c139c3103ac9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ac5ece4-dabf-4248-892e-c139c3103ac9-dshm\") pod \"5ac5ece4-dabf-4248-892e-c139c3103ac9\" (UID: \"5ac5ece4-dabf-4248-892e-c139c3103ac9\") " Nov 01 20:40:48.024701 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:40:48.024620 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a9fbeb6-0f00-4a8c-9d76-f7550dd55775-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a9fbeb6-0f00-4a8c-9d76-f7550dd55775-tmp-dir\") pod \"5a9fbeb6-0f00-4a8c-9d76-f7550dd55775\" (UID: \"5a9fbeb6-0f00-4a8c-9d76-f7550dd55775\") " Nov 01 20:40:48.024701 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:40:48.024659 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ac5ece4-dabf-4248-892e-c139c3103ac9-kube-api-access-gfk2b\" (UniqueName: \"kubernetes.io/projected/5ac5ece4-dabf-4248-892e-c139c3103ac9-kube-api-access-gfk2b\") pod \"5ac5ece4-dabf-4248-892e-c139c3103ac9\" (UID: \"5ac5ece4-dabf-4248-892e-c139c3103ac9\") " Nov 01 20:40:48.024701 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:40:48.024687 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ac5ece4-dabf-4248-892e-c139c3103ac9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ac5ece4-dabf-4248-892e-c139c3103ac9-tmp-dir\") pod \"5ac5ece4-dabf-4248-892e-c139c3103ac9\" (UID: \"5ac5ece4-dabf-4248-892e-c139c3103ac9\") " Nov 01 20:40:48.024701 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:40:48.024618 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a9fbeb6-0f00-4a8c-9d76-f7550dd55775-home" pod "5a9fbeb6-0f00-4a8c-9d76-f7550dd55775" (UID: "5a9fbeb6-0f00-4a8c-9d76-f7550dd55775"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:40:48.024965 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:40:48.024737 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ac5ece4-dabf-4248-892e-c139c3103ac9-home" pod "5ac5ece4-dabf-4248-892e-c139c3103ac9" (UID: "5ac5ece4-dabf-4248-892e-c139c3103ac9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:40:48.024965 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:40:48.024773 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a9fbeb6-0f00-4a8c-9d76-f7550dd55775-model-cache" pod "5a9fbeb6-0f00-4a8c-9d76-f7550dd55775" (UID: "5a9fbeb6-0f00-4a8c-9d76-f7550dd55775"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:40:48.024965 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:40:48.024813 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a9fbeb6-0f00-4a8c-9d76-f7550dd55775-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a9fbeb6-0f00-4a8c-9d76-f7550dd55775-dshm\") pod \"5a9fbeb6-0f00-4a8c-9d76-f7550dd55775\" (UID: \"5a9fbeb6-0f00-4a8c-9d76-f7550dd55775\") " Nov 01 20:40:48.024965 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:40:48.024858 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ac5ece4-dabf-4248-892e-c139c3103ac9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ac5ece4-dabf-4248-892e-c139c3103ac9-model-cache\") pod \"5ac5ece4-dabf-4248-892e-c139c3103ac9\" (UID: \"5ac5ece4-dabf-4248-892e-c139c3103ac9\") " Nov 01 20:40:48.024965 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:40:48.024872 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a9fbeb6-0f00-4a8c-9d76-f7550dd55775-tmp-dir" pod "5a9fbeb6-0f00-4a8c-9d76-f7550dd55775" (UID: "5a9fbeb6-0f00-4a8c-9d76-f7550dd55775"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:40:48.024965 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:40:48.024895 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a9fbeb6-0f00-4a8c-9d76-f7550dd55775-kube-api-access-4ksww\" (UniqueName: \"kubernetes.io/projected/5a9fbeb6-0f00-4a8c-9d76-f7550dd55775-kube-api-access-4ksww\") pod \"5a9fbeb6-0f00-4a8c-9d76-f7550dd55775\" (UID: \"5a9fbeb6-0f00-4a8c-9d76-f7550dd55775\") " Nov 01 20:40:48.024965 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:40:48.024914 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ac5ece4-dabf-4248-892e-c139c3103ac9-tmp-dir" pod "5ac5ece4-dabf-4248-892e-c139c3103ac9" (UID: "5ac5ece4-dabf-4248-892e-c139c3103ac9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:40:48.025422 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:40:48.024965 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a9fbeb6-0f00-4a8c-9d76-f7550dd55775-tls-certs\" (UniqueName: \"kubernetes.io/secret/5a9fbeb6-0f00-4a8c-9d76-f7550dd55775-tls-certs\") pod \"5a9fbeb6-0f00-4a8c-9d76-f7550dd55775\" (UID: \"5a9fbeb6-0f00-4a8c-9d76-f7550dd55775\") " Nov 01 20:40:48.025422 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:40:48.025093 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ac5ece4-dabf-4248-892e-c139c3103ac9-model-cache" pod "5ac5ece4-dabf-4248-892e-c139c3103ac9" (UID: "5ac5ece4-dabf-4248-892e-c139c3103ac9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:40:48.025422 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:40:48.025383 5935 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ac5ece4-dabf-4248-892e-c139c3103ac9-home\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:40:48.025422 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:40:48.025406 5935 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a9fbeb6-0f00-4a8c-9d76-f7550dd55775-model-cache\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:40:48.025422 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:40:48.025421 5935 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a9fbeb6-0f00-4a8c-9d76-f7550dd55775-tmp-dir\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:40:48.025721 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:40:48.025436 5935 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ac5ece4-dabf-4248-892e-c139c3103ac9-tmp-dir\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:40:48.025721 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:40:48.025448 5935 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ac5ece4-dabf-4248-892e-c139c3103ac9-model-cache\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:40:48.025721 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:40:48.025461 5935 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a9fbeb6-0f00-4a8c-9d76-f7550dd55775-home\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:40:48.026976 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:40:48.026943 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a9fbeb6-0f00-4a8c-9d76-f7550dd55775-dshm" pod "5a9fbeb6-0f00-4a8c-9d76-f7550dd55775" (UID: "5a9fbeb6-0f00-4a8c-9d76-f7550dd55775"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:40:48.027106 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:40:48.027079 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a9fbeb6-0f00-4a8c-9d76-f7550dd55775-kube-api-access-4ksww" pod "5a9fbeb6-0f00-4a8c-9d76-f7550dd55775" (UID: "5a9fbeb6-0f00-4a8c-9d76-f7550dd55775"). InnerVolumeSpecName "kube-api-access-4ksww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:40:48.027106 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:40:48.027097 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ac5ece4-dabf-4248-892e-c139c3103ac9-dshm" pod "5ac5ece4-dabf-4248-892e-c139c3103ac9" (UID: "5ac5ece4-dabf-4248-892e-c139c3103ac9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:40:48.027193 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:40:48.027108 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ac5ece4-dabf-4248-892e-c139c3103ac9-tls-certs" pod "5ac5ece4-dabf-4248-892e-c139c3103ac9" (UID: "5ac5ece4-dabf-4248-892e-c139c3103ac9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:40:48.027232 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:40:48.027196 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ac5ece4-dabf-4248-892e-c139c3103ac9-kube-api-access-gfk2b" pod "5ac5ece4-dabf-4248-892e-c139c3103ac9" (UID: "5ac5ece4-dabf-4248-892e-c139c3103ac9"). InnerVolumeSpecName "kube-api-access-gfk2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:40:48.027344 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:40:48.027330 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a9fbeb6-0f00-4a8c-9d76-f7550dd55775-tls-certs" pod "5a9fbeb6-0f00-4a8c-9d76-f7550dd55775" (UID: "5a9fbeb6-0f00-4a8c-9d76-f7550dd55775"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:40:48.125914 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:40:48.125885 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfk2b\" (UniqueName: \"kubernetes.io/projected/5ac5ece4-dabf-4248-892e-c139c3103ac9-kube-api-access-gfk2b\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:40:48.125914 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:40:48.125912 5935 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a9fbeb6-0f00-4a8c-9d76-f7550dd55775-dshm\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:40:48.125914 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:40:48.125921 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ksww\" (UniqueName: \"kubernetes.io/projected/5a9fbeb6-0f00-4a8c-9d76-f7550dd55775-kube-api-access-4ksww\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:40:48.126140 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:40:48.125930 5935 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a9fbeb6-0f00-4a8c-9d76-f7550dd55775-tls-certs\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:40:48.126140 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:40:48.125939 5935 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ac5ece4-dabf-4248-892e-c139c3103ac9-tls-certs\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:40:48.126140 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:40:48.125948 5935 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ac5ece4-dabf-4248-892e-c139c3103ac9-dshm\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:40:48.900438 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:40:48.900398 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-main-1bfa0e/simulator-main-1bfa0e-kserve-847dc858bb-k46l8" event={"ID":"5a9fbeb6-0f00-4a8c-9d76-f7550dd55775","Type":"ContainerDied","Data":"972afc34b75308d90951edba6b0f8853e381b6429ab641aae9e7ffcffe270d0b"} Nov 01 20:40:48.900438 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:40:48.900424 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-main-1bfa0e/premium-main-1bfa0e-kserve-546887c779-lbfjg" Nov 01 20:40:48.900921 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:40:48.900451 5935 scope.go:122] "RemoveContainer" containerID="0cba2a7b9e9c49e590a6b60aa58f9f4d1c4d9020bb003da8c474bba8c49e5f37" Nov 01 20:40:48.900921 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:40:48.900424 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-main-1bfa0e/simulator-main-1bfa0e-kserve-847dc858bb-k46l8" Nov 01 20:40:48.921751 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:40:48.921726 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-main-1bfa0e/premium-main-1bfa0e-kserve-546887c779-lbfjg"] Nov 01 20:40:48.924070 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:40:48.924037 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-main-1bfa0e/premium-main-1bfa0e-kserve-546887c779-lbfjg"] Nov 01 20:40:48.933405 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:40:48.933386 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-main-1bfa0e/simulator-main-1bfa0e-kserve-847dc858bb-k46l8"] Nov 01 20:40:48.937084 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:40:48.937063 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-main-1bfa0e/simulator-main-1bfa0e-kserve-847dc858bb-k46l8"] Nov 01 20:40:50.352033 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:40:50.352002 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a9fbeb6-0f00-4a8c-9d76-f7550dd55775" path="/var/lib/kubelet/pods/5a9fbeb6-0f00-4a8c-9d76-f7550dd55775/volumes" Nov 01 20:40:50.352582 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:40:50.352559 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ac5ece4-dabf-4248-892e-c139c3103ac9" path="/var/lib/kubelet/pods/5ac5ece4-dabf-4248-892e-c139c3103ac9/volumes" Nov 01 20:41:05.843500 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:41:05.843463 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:41:08.280388 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:08.280355 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb"] Nov 01 20:41:08.280765 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:08.280711 5935 cpu_manager.go:414] "containerMap: removing container" podUID="5ac5ece4-dabf-4248-892e-c139c3103ac9" containerName="main" Nov 01 20:41:08.280765 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:08.280722 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ac5ece4-dabf-4248-892e-c139c3103ac9" containerName="main" Nov 01 20:41:08.280765 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:08.280741 5935 cpu_manager.go:414] "containerMap: removing container" podUID="38a5cb76-70ed-4e9e-b6c8-a4e794f500cc" containerName="revoke-keys" Nov 01 20:41:08.280765 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:08.280746 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38a5cb76-70ed-4e9e-b6c8-a4e794f500cc" containerName="revoke-keys" Nov 01 20:41:08.280765 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:08.280761 5935 cpu_manager.go:414] "containerMap: removing container" podUID="5a9fbeb6-0f00-4a8c-9d76-f7550dd55775" containerName="main" Nov 01 20:41:08.280765 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:08.280766 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a9fbeb6-0f00-4a8c-9d76-f7550dd55775" containerName="main" Nov 01 20:41:08.280955 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:08.280817 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a9fbeb6-0f00-4a8c-9d76-f7550dd55775" containerName="main" Nov 01 20:41:08.280955 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:08.280828 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ac5ece4-dabf-4248-892e-c139c3103ac9" containerName="main" Nov 01 20:41:08.280955 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:08.280836 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="38a5cb76-70ed-4e9e-b6c8-a4e794f500cc" containerName="revoke-keys" Nov 01 20:41:08.285424 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:08.285404 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb" Nov 01 20:41:08.292805 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:08.292784 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-main-1bfa0e-openshift-default-dockercfg-hbng5\"" Nov 01 20:41:08.296193 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:08.296171 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb"] Nov 01 20:41:08.306744 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:08.306712 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fdd0ab2-37ef-4983-84a2-87eefed228a3-istiod-ca-cert\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb\" (UID: \"3fdd0ab2-37ef-4983-84a2-87eefed228a3\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb" Nov 01 20:41:08.306852 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:08.306811 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fdd0ab2-37ef-4983-84a2-87eefed228a3-istio-podinfo\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb\" (UID: \"3fdd0ab2-37ef-4983-84a2-87eefed228a3\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb" Nov 01 20:41:08.306852 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:08.306843 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fdd0ab2-37ef-4983-84a2-87eefed228a3-istio-token\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb\" (UID: \"3fdd0ab2-37ef-4983-84a2-87eefed228a3\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb" Nov 01 20:41:08.306936 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:08.306874 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrfs6\" (UniqueName: \"kubernetes.io/projected/3fdd0ab2-37ef-4983-84a2-87eefed228a3-kube-api-access-qrfs6\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb\" (UID: \"3fdd0ab2-37ef-4983-84a2-87eefed228a3\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb" Nov 01 20:41:08.306936 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:08.306901 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fdd0ab2-37ef-4983-84a2-87eefed228a3-istio-envoy\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb\" (UID: \"3fdd0ab2-37ef-4983-84a2-87eefed228a3\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb" Nov 01 20:41:08.306936 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:08.306929 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fdd0ab2-37ef-4983-84a2-87eefed228a3-workload-certs\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb\" (UID: \"3fdd0ab2-37ef-4983-84a2-87eefed228a3\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb" Nov 01 20:41:08.307039 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:08.306986 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fdd0ab2-37ef-4983-84a2-87eefed228a3-credential-socket\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb\" (UID: \"3fdd0ab2-37ef-4983-84a2-87eefed228a3\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb" Nov 01 20:41:08.307039 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:08.307023 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fdd0ab2-37ef-4983-84a2-87eefed228a3-istio-data\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb\" (UID: \"3fdd0ab2-37ef-4983-84a2-87eefed228a3\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb" Nov 01 20:41:08.307103 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:08.307049 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fdd0ab2-37ef-4983-84a2-87eefed228a3-workload-socket\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb\" (UID: \"3fdd0ab2-37ef-4983-84a2-87eefed228a3\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb" Nov 01 20:41:08.408094 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:08.408064 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fdd0ab2-37ef-4983-84a2-87eefed228a3-istio-podinfo\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb\" (UID: \"3fdd0ab2-37ef-4983-84a2-87eefed228a3\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb" Nov 01 20:41:08.408264 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:08.408102 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fdd0ab2-37ef-4983-84a2-87eefed228a3-istio-token\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb\" (UID: \"3fdd0ab2-37ef-4983-84a2-87eefed228a3\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb" Nov 01 20:41:08.408264 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:08.408129 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrfs6\" (UniqueName: \"kubernetes.io/projected/3fdd0ab2-37ef-4983-84a2-87eefed228a3-kube-api-access-qrfs6\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb\" (UID: \"3fdd0ab2-37ef-4983-84a2-87eefed228a3\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb" Nov 01 20:41:08.408384 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:08.408365 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fdd0ab2-37ef-4983-84a2-87eefed228a3-istio-envoy\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb\" (UID: \"3fdd0ab2-37ef-4983-84a2-87eefed228a3\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb" Nov 01 20:41:08.408439 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:08.408402 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fdd0ab2-37ef-4983-84a2-87eefed228a3-workload-certs\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb\" (UID: \"3fdd0ab2-37ef-4983-84a2-87eefed228a3\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb" Nov 01 20:41:08.408488 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:08.408473 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fdd0ab2-37ef-4983-84a2-87eefed228a3-credential-socket\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb\" (UID: \"3fdd0ab2-37ef-4983-84a2-87eefed228a3\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb" Nov 01 20:41:08.408534 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:08.408512 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fdd0ab2-37ef-4983-84a2-87eefed228a3-istio-data\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb\" (UID: \"3fdd0ab2-37ef-4983-84a2-87eefed228a3\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb" Nov 01 20:41:08.408584 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:08.408538 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fdd0ab2-37ef-4983-84a2-87eefed228a3-workload-socket\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb\" (UID: \"3fdd0ab2-37ef-4983-84a2-87eefed228a3\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb" Nov 01 20:41:08.408621 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:08.408588 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fdd0ab2-37ef-4983-84a2-87eefed228a3-istiod-ca-cert\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb\" (UID: \"3fdd0ab2-37ef-4983-84a2-87eefed228a3\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb" Nov 01 20:41:08.408883 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:08.408855 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fdd0ab2-37ef-4983-84a2-87eefed228a3-workload-certs\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb\" (UID: \"3fdd0ab2-37ef-4983-84a2-87eefed228a3\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb" Nov 01 20:41:08.409020 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:08.408883 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fdd0ab2-37ef-4983-84a2-87eefed228a3-credential-socket\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb\" (UID: \"3fdd0ab2-37ef-4983-84a2-87eefed228a3\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb" Nov 01 20:41:08.409020 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:08.408935 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fdd0ab2-37ef-4983-84a2-87eefed228a3-istio-data\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb\" (UID: \"3fdd0ab2-37ef-4983-84a2-87eefed228a3\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb" Nov 01 20:41:08.409020 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:08.409005 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fdd0ab2-37ef-4983-84a2-87eefed228a3-workload-socket\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb\" (UID: \"3fdd0ab2-37ef-4983-84a2-87eefed228a3\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb" Nov 01 20:41:08.409273 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:08.409256 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fdd0ab2-37ef-4983-84a2-87eefed228a3-istiod-ca-cert\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb\" (UID: \"3fdd0ab2-37ef-4983-84a2-87eefed228a3\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb" Nov 01 20:41:08.410547 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:08.410527 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fdd0ab2-37ef-4983-84a2-87eefed228a3-istio-envoy\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb\" (UID: \"3fdd0ab2-37ef-4983-84a2-87eefed228a3\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb" Nov 01 20:41:08.410693 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:08.410677 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fdd0ab2-37ef-4983-84a2-87eefed228a3-istio-podinfo\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb\" (UID: \"3fdd0ab2-37ef-4983-84a2-87eefed228a3\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb" Nov 01 20:41:08.415768 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:08.415747 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrfs6\" (UniqueName: \"kubernetes.io/projected/3fdd0ab2-37ef-4983-84a2-87eefed228a3-kube-api-access-qrfs6\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb\" (UID: \"3fdd0ab2-37ef-4983-84a2-87eefed228a3\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb" Nov 01 20:41:08.415902 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:08.415882 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fdd0ab2-37ef-4983-84a2-87eefed228a3-istio-token\") pod \"e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb\" (UID: \"3fdd0ab2-37ef-4983-84a2-87eefed228a3\") " pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb" Nov 01 20:41:08.599066 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:08.598973 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb" Nov 01 20:41:08.872092 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:08.872060 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb"] Nov 01 20:41:08.874146 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:41:08.874122 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fdd0ab2_37ef_4983_84a2_87eefed228a3.slice/crio-fea456f2ac084deddbdc64204be575f9c4209f90f4151584bf85cc60507e755c WatchSource:0}: Error finding container fea456f2ac084deddbdc64204be575f9c4209f90f4151584bf85cc60507e755c: Status 404 returned error can't find the container with id fea456f2ac084deddbdc64204be575f9c4209f90f4151584bf85cc60507e755c Nov 01 20:41:08.876741 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:08.876713 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:41:08.876816 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:08.876778 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:41:08.876816 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:08.876806 5935 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Nov 01 20:41:09.003818 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:09.003783 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb" event={"ID":"3fdd0ab2-37ef-4983-84a2-87eefed228a3","Type":"ContainerStarted","Data":"ca72ca22f73d6b61b146b38a3d28da118fbebe15cede99d6980a8e56e1fb6c5f"} Nov 01 20:41:09.003988 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:09.003862 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb" event={"ID":"3fdd0ab2-37ef-4983-84a2-87eefed228a3","Type":"ContainerStarted","Data":"fea456f2ac084deddbdc64204be575f9c4209f90f4151584bf85cc60507e755c"} Nov 01 20:41:09.022578 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:09.022531 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb" podStartSLOduration=1.022515651 podStartE2EDuration="1.022515651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:41:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:41:09.021264733 +0000 UTC m=+2514.794425247" watchObservedRunningTime="2026-11-01 20:41:09.022515651 +0000 UTC m=+2514.795676162" Nov 01 20:41:09.599480 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:09.599448 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb" Nov 01 20:41:09.600970 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:09.600941 5935 patch_prober.go:29] interesting pod/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 01 20:41:09.601113 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:09.600997 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb" podUID="3fdd0ab2-37ef-4983-84a2-87eefed228a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 01 20:41:10.599416 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:10.599376 5935 patch_prober.go:29] interesting pod/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 01 20:41:10.599594 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:10.599433 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb" podUID="3fdd0ab2-37ef-4983-84a2-87eefed228a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 01 20:41:11.421539 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:11.421496 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb"] Nov 01 20:41:11.600109 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:11.600072 5935 patch_prober.go:29] interesting pod/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Nov 01 20:41:11.600534 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:11.600126 5935 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb" podUID="3fdd0ab2-37ef-4983-84a2-87eefed228a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Nov 01 20:41:12.023009 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:12.022971 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb" podUID="3fdd0ab2-37ef-4983-84a2-87eefed228a3" containerName="istio-proxy" containerID="cri-o://ca72ca22f73d6b61b146b38a3d28da118fbebe15cede99d6980a8e56e1fb6c5f" gracePeriod=30 Nov 01 20:41:15.208069 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:15.208034 5935 scope.go:122] "RemoveContainer" containerID="72316667f4213f1440f02583d8d28880e3e97814c09fc80da31de6d7804be4e4" Nov 01 20:41:20.065105 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:20.065068 5935 generic.go:356] "Generic (PLEG): container finished" podID="3fdd0ab2-37ef-4983-84a2-87eefed228a3" containerID="ca72ca22f73d6b61b146b38a3d28da118fbebe15cede99d6980a8e56e1fb6c5f" exitCode=0 Nov 01 20:41:20.065630 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:20.065223 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb" event={"ID":"3fdd0ab2-37ef-4983-84a2-87eefed228a3","Type":"ContainerDied","Data":"ca72ca22f73d6b61b146b38a3d28da118fbebe15cede99d6980a8e56e1fb6c5f"} Nov 01 20:41:20.161159 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:20.161130 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb" Nov 01 20:41:20.241929 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:20.241847 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fdd0ab2-37ef-4983-84a2-87eefed228a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fdd0ab2-37ef-4983-84a2-87eefed228a3-workload-socket\") pod \"3fdd0ab2-37ef-4983-84a2-87eefed228a3\" (UID: \"3fdd0ab2-37ef-4983-84a2-87eefed228a3\") " Nov 01 20:41:20.241929 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:20.241895 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fdd0ab2-37ef-4983-84a2-87eefed228a3-istio-token\" (UniqueName: \"kubernetes.io/projected/3fdd0ab2-37ef-4983-84a2-87eefed228a3-istio-token\") pod \"3fdd0ab2-37ef-4983-84a2-87eefed228a3\" (UID: \"3fdd0ab2-37ef-4983-84a2-87eefed228a3\") " Nov 01 20:41:20.241929 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:20.241919 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fdd0ab2-37ef-4983-84a2-87eefed228a3-kube-api-access-qrfs6\" (UniqueName: \"kubernetes.io/projected/3fdd0ab2-37ef-4983-84a2-87eefed228a3-kube-api-access-qrfs6\") pod \"3fdd0ab2-37ef-4983-84a2-87eefed228a3\" (UID: \"3fdd0ab2-37ef-4983-84a2-87eefed228a3\") " Nov 01 20:41:20.242209 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:20.241959 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fdd0ab2-37ef-4983-84a2-87eefed228a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fdd0ab2-37ef-4983-84a2-87eefed228a3-istio-envoy\") pod \"3fdd0ab2-37ef-4983-84a2-87eefed228a3\" (UID: \"3fdd0ab2-37ef-4983-84a2-87eefed228a3\") " Nov 01 20:41:20.242209 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:20.241988 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fdd0ab2-37ef-4983-84a2-87eefed228a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fdd0ab2-37ef-4983-84a2-87eefed228a3-credential-socket\") pod \"3fdd0ab2-37ef-4983-84a2-87eefed228a3\" (UID: \"3fdd0ab2-37ef-4983-84a2-87eefed228a3\") " Nov 01 20:41:20.242209 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:20.242067 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3fdd0ab2-37ef-4983-84a2-87eefed228a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fdd0ab2-37ef-4983-84a2-87eefed228a3-istio-podinfo\") pod \"3fdd0ab2-37ef-4983-84a2-87eefed228a3\" (UID: \"3fdd0ab2-37ef-4983-84a2-87eefed228a3\") " Nov 01 20:41:20.242209 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:20.242094 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fdd0ab2-37ef-4983-84a2-87eefed228a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fdd0ab2-37ef-4983-84a2-87eefed228a3-istio-data\") pod \"3fdd0ab2-37ef-4983-84a2-87eefed228a3\" (UID: \"3fdd0ab2-37ef-4983-84a2-87eefed228a3\") " Nov 01 20:41:20.242209 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:20.242168 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fdd0ab2-37ef-4983-84a2-87eefed228a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fdd0ab2-37ef-4983-84a2-87eefed228a3-workload-certs\") pod \"3fdd0ab2-37ef-4983-84a2-87eefed228a3\" (UID: \"3fdd0ab2-37ef-4983-84a2-87eefed228a3\") " Nov 01 20:41:20.242209 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:20.242177 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fdd0ab2-37ef-4983-84a2-87eefed228a3-workload-socket" pod "3fdd0ab2-37ef-4983-84a2-87eefed228a3" (UID: "3fdd0ab2-37ef-4983-84a2-87eefed228a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:41:20.242580 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:20.242227 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fdd0ab2-37ef-4983-84a2-87eefed228a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fdd0ab2-37ef-4983-84a2-87eefed228a3-istiod-ca-cert\") pod \"3fdd0ab2-37ef-4983-84a2-87eefed228a3\" (UID: \"3fdd0ab2-37ef-4983-84a2-87eefed228a3\") " Nov 01 20:41:20.242580 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:20.242385 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fdd0ab2-37ef-4983-84a2-87eefed228a3-credential-socket" pod "3fdd0ab2-37ef-4983-84a2-87eefed228a3" (UID: "3fdd0ab2-37ef-4983-84a2-87eefed228a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:41:20.242580 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:20.242410 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fdd0ab2-37ef-4983-84a2-87eefed228a3-istio-data" pod "3fdd0ab2-37ef-4983-84a2-87eefed228a3" (UID: "3fdd0ab2-37ef-4983-84a2-87eefed228a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:41:20.242580 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:20.242569 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fdd0ab2-37ef-4983-84a2-87eefed228a3-workload-certs" pod "3fdd0ab2-37ef-4983-84a2-87eefed228a3" (UID: "3fdd0ab2-37ef-4983-84a2-87eefed228a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:41:20.242798 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:20.242733 5935 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fdd0ab2-37ef-4983-84a2-87eefed228a3-istio-data\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:41:20.242798 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:20.242752 5935 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fdd0ab2-37ef-4983-84a2-87eefed228a3-workload-certs\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:41:20.242798 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:20.242755 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fdd0ab2-37ef-4983-84a2-87eefed228a3-istiod-ca-cert" pod "3fdd0ab2-37ef-4983-84a2-87eefed228a3" (UID: "3fdd0ab2-37ef-4983-84a2-87eefed228a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:41:20.242798 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:20.242766 5935 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fdd0ab2-37ef-4983-84a2-87eefed228a3-workload-socket\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:41:20.242798 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:20.242780 5935 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fdd0ab2-37ef-4983-84a2-87eefed228a3-credential-socket\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:41:20.244142 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:20.244112 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3fdd0ab2-37ef-4983-84a2-87eefed228a3-istio-podinfo" pod "3fdd0ab2-37ef-4983-84a2-87eefed228a3" (UID: "3fdd0ab2-37ef-4983-84a2-87eefed228a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 01 20:41:20.244313 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:20.244149 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fdd0ab2-37ef-4983-84a2-87eefed228a3-istio-token" pod "3fdd0ab2-37ef-4983-84a2-87eefed228a3" (UID: "3fdd0ab2-37ef-4983-84a2-87eefed228a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:41:20.244313 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:20.244243 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fdd0ab2-37ef-4983-84a2-87eefed228a3-istio-envoy" pod "3fdd0ab2-37ef-4983-84a2-87eefed228a3" (UID: "3fdd0ab2-37ef-4983-84a2-87eefed228a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:41:20.244567 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:20.244549 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fdd0ab2-37ef-4983-84a2-87eefed228a3-kube-api-access-qrfs6" pod "3fdd0ab2-37ef-4983-84a2-87eefed228a3" (UID: "3fdd0ab2-37ef-4983-84a2-87eefed228a3"). InnerVolumeSpecName "kube-api-access-qrfs6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:41:20.343998 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:20.343973 5935 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fdd0ab2-37ef-4983-84a2-87eefed228a3-istio-token\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:41:20.343998 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:20.343994 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrfs6\" (UniqueName: \"kubernetes.io/projected/3fdd0ab2-37ef-4983-84a2-87eefed228a3-kube-api-access-qrfs6\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:41:20.344149 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:20.344005 5935 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fdd0ab2-37ef-4983-84a2-87eefed228a3-istio-envoy\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:41:20.344149 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:20.344016 5935 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fdd0ab2-37ef-4983-84a2-87eefed228a3-istio-podinfo\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:41:20.344149 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:20.344025 5935 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fdd0ab2-37ef-4983-84a2-87eefed228a3-istiod-ca-cert\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:41:21.071458 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:21.071424 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb" Nov 01 20:41:21.071458 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:21.071443 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb" event={"ID":"3fdd0ab2-37ef-4983-84a2-87eefed228a3","Type":"ContainerDied","Data":"fea456f2ac084deddbdc64204be575f9c4209f90f4151584bf85cc60507e755c"} Nov 01 20:41:21.071928 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:21.071542 5935 scope.go:122] "RemoveContainer" containerID="ca72ca22f73d6b61b146b38a3d28da118fbebe15cede99d6980a8e56e1fb6c5f" Nov 01 20:41:21.088530 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:21.088503 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb"] Nov 01 20:41:21.091327 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:21.091301 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-main-1bfa0e-openshift-default-66f45d6dff-r9mtb"] Nov 01 20:41:22.352475 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:41:22.352443 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fdd0ab2-37ef-4983-84a2-87eefed228a3" path="/var/lib/kubelet/pods/3fdd0ab2-37ef-4983-84a2-87eefed228a3/volumes" Nov 01 20:41:35.843978 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:41:35.843871 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:42:05.843191 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:42:05.843148 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:42:11.709484 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:42:11.709419 5935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-ljtcq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 20:42:11.710200 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:42:11.710156 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-ljtcq" podUID="03014616-4520-48bb-a548-66485577e26c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:42:17.605311 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:42:17.605110 5935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-m6qv8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 20:42:17.605311 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:42:17.605240 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6qv8" podUID="9de91381-a29d-422d-b859-69229e8970c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:42:17.815880 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:42:17.815402 5935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-fl2rn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 20:42:17.815880 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:42:17.815471 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-fl2rn" podUID="043fcc04-22eb-454f-a1d5-d669214c5b0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:42:35.843003 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:42:35.842969 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:42:53.018593 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:42:53.018558 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-fl2rj"] Nov 01 20:42:53.021163 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:42:53.021141 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-fl2rj"] Nov 01 20:42:54.351937 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:42:54.351907 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38a5cb76-70ed-4e9e-b6c8-a4e794f500cc" path="/var/lib/kubelet/pods/38a5cb76-70ed-4e9e-b6c8-a4e794f500cc/volumes" Nov 01 20:43:02.604318 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:43:02.603481 5935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-m6qv8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 20:43:02.604318 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:43:02.603556 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6qv8" podUID="9de91381-a29d-422d-b859-69229e8970c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:43:02.808495 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:43:02.808444 5935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-fl2rn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 20:43:02.808837 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:43:02.808806 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-fl2rn" podUID="043fcc04-22eb-454f-a1d5-d669214c5b0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:43:05.843093 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:43:05.843054 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:43:35.842975 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:43:35.842940 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:44:05.842980 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:44:05.842942 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:44:15.259721 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:44:15.259667 5935 scope.go:122] "RemoveContainer" containerID="84b94ba4bd0478a31b2b6fa8cbcb0c9d10b1d1bb07ad344e55b22bffab53f0e4" Nov 01 20:44:35.843349 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:44:35.843253 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:45:00.129571 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:00.129533 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892765-8fxfg"] Nov 01 20:45:00.130098 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:00.129897 5935 cpu_manager.go:414] "containerMap: removing container" podUID="3fdd0ab2-37ef-4983-84a2-87eefed228a3" containerName="istio-proxy" Nov 01 20:45:00.130098 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:00.129908 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fdd0ab2-37ef-4983-84a2-87eefed228a3" containerName="istio-proxy" Nov 01 20:45:00.130098 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:00.129991 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fdd0ab2-37ef-4983-84a2-87eefed228a3" containerName="istio-proxy" Nov 01 20:45:00.133086 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:00.133070 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892765-8fxfg" Nov 01 20:45:00.138719 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:00.138688 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6rcz6\"" Nov 01 20:45:00.140202 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:00.140180 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892765-8fxfg"] Nov 01 20:45:00.206417 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:00.206375 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jccb8\" (UniqueName: \"kubernetes.io/projected/8d710b1f-2bab-406e-9e4c-6e39672284d8-kube-api-access-jccb8\") pod \"maas-api-key-cleanup-29892765-8fxfg\" (UID: \"8d710b1f-2bab-406e-9e4c-6e39672284d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892765-8fxfg" Nov 01 20:45:00.206744 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:00.206720 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d710b1f-2bab-406e-9e4c-6e39672284d8-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892765-8fxfg\" (UID: \"8d710b1f-2bab-406e-9e4c-6e39672284d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892765-8fxfg" Nov 01 20:45:00.310535 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:00.310502 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jccb8\" (UniqueName: \"kubernetes.io/projected/8d710b1f-2bab-406e-9e4c-6e39672284d8-kube-api-access-jccb8\") pod \"maas-api-key-cleanup-29892765-8fxfg\" (UID: \"8d710b1f-2bab-406e-9e4c-6e39672284d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892765-8fxfg" Nov 01 20:45:00.310916 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:00.310884 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d710b1f-2bab-406e-9e4c-6e39672284d8-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892765-8fxfg\" (UID: \"8d710b1f-2bab-406e-9e4c-6e39672284d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892765-8fxfg" Nov 01 20:45:00.311982 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:00.311866 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d710b1f-2bab-406e-9e4c-6e39672284d8-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892765-8fxfg\" (UID: \"8d710b1f-2bab-406e-9e4c-6e39672284d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892765-8fxfg" Nov 01 20:45:00.318878 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:00.318830 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jccb8\" (UniqueName: \"kubernetes.io/projected/8d710b1f-2bab-406e-9e4c-6e39672284d8-kube-api-access-jccb8\") pod \"maas-api-key-cleanup-29892765-8fxfg\" (UID: \"8d710b1f-2bab-406e-9e4c-6e39672284d8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892765-8fxfg" Nov 01 20:45:00.473314 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:00.471409 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892765-8fxfg" Nov 01 20:45:00.882336 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:00.882187 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892765-8fxfg"] Nov 01 20:45:00.901691 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:45:00.901634 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d710b1f_2bab_406e_9e4c_6e39672284d8.slice/crio-997748c8375844e10781631a741409655011d31b7701da31a79b483c5c01a83f WatchSource:0}: Error finding container 997748c8375844e10781631a741409655011d31b7701da31a79b483c5c01a83f: Status 404 returned error can't find the container with id 997748c8375844e10781631a741409655011d31b7701da31a79b483c5c01a83f Nov 01 20:45:01.323569 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:01.323528 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892765-8fxfg" event={"ID":"8d710b1f-2bab-406e-9e4c-6e39672284d8","Type":"ContainerStarted","Data":"997748c8375844e10781631a741409655011d31b7701da31a79b483c5c01a83f"} Nov 01 20:45:02.329857 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:02.329770 5935 generic.go:356] "Generic (PLEG): container finished" podID="8d710b1f-2bab-406e-9e4c-6e39672284d8" containerID="26e09ba18970a80f1683e5682e51ed697782af8f92b9ca7223af00c4fd87fed7" exitCode=60 Nov 01 20:45:02.330232 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:02.329858 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892765-8fxfg" event={"ID":"8d710b1f-2bab-406e-9e4c-6e39672284d8","Type":"ContainerDied","Data":"26e09ba18970a80f1683e5682e51ed697782af8f92b9ca7223af00c4fd87fed7"} Nov 01 20:45:02.330232 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:02.330031 5935 scope.go:122] "RemoveContainer" containerID="26e09ba18970a80f1683e5682e51ed697782af8f92b9ca7223af00c4fd87fed7" Nov 01 20:45:03.336208 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:03.336116 5935 generic.go:356] "Generic (PLEG): container finished" podID="8d710b1f-2bab-406e-9e4c-6e39672284d8" containerID="a2ee6df89d878ca2b93f4999f7d9bda0e436dad050d8287a65000201ce1fd52f" exitCode=60 Nov 01 20:45:03.336626 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:03.336217 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892765-8fxfg" event={"ID":"8d710b1f-2bab-406e-9e4c-6e39672284d8","Type":"ContainerDied","Data":"a2ee6df89d878ca2b93f4999f7d9bda0e436dad050d8287a65000201ce1fd52f"} Nov 01 20:45:03.336626 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:03.336259 5935 scope.go:122] "RemoveContainer" containerID="26e09ba18970a80f1683e5682e51ed697782af8f92b9ca7223af00c4fd87fed7" Nov 01 20:45:03.336626 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:03.336499 5935 scope.go:122] "RemoveContainer" containerID="a2ee6df89d878ca2b93f4999f7d9bda0e436dad050d8287a65000201ce1fd52f" Nov 01 20:45:03.336768 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:45:03.336712 5935 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892765-8fxfg_odh-ai-gateway-infra(8d710b1f-2bab-406e-9e4c-6e39672284d8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892765-8fxfg" podUID="8d710b1f-2bab-406e-9e4c-6e39672284d8" Nov 01 20:45:04.343414 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:04.343384 5935 scope.go:122] "RemoveContainer" containerID="a2ee6df89d878ca2b93f4999f7d9bda0e436dad050d8287a65000201ce1fd52f" Nov 01 20:45:04.343809 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:45:04.343591 5935 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892765-8fxfg_odh-ai-gateway-infra(8d710b1f-2bab-406e-9e4c-6e39672284d8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892765-8fxfg" podUID="8d710b1f-2bab-406e-9e4c-6e39672284d8" Nov 01 20:45:05.842869 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:45:05.842836 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:45:15.346154 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:15.346122 5935 scope.go:122] "RemoveContainer" containerID="a2ee6df89d878ca2b93f4999f7d9bda0e436dad050d8287a65000201ce1fd52f" Nov 01 20:45:16.398082 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:16.398000 5935 generic.go:356] "Generic (PLEG): container finished" podID="8d710b1f-2bab-406e-9e4c-6e39672284d8" containerID="8c2e3d02799dd914f0c8a8e4513791276a208976e1a1a7bc48ffc3d73d2cdf3e" exitCode=60 Nov 01 20:45:16.398082 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:16.398055 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892765-8fxfg" event={"ID":"8d710b1f-2bab-406e-9e4c-6e39672284d8","Type":"ContainerDied","Data":"8c2e3d02799dd914f0c8a8e4513791276a208976e1a1a7bc48ffc3d73d2cdf3e"} Nov 01 20:45:16.398534 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:16.398101 5935 scope.go:122] "RemoveContainer" containerID="a2ee6df89d878ca2b93f4999f7d9bda0e436dad050d8287a65000201ce1fd52f" Nov 01 20:45:16.398534 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:16.398408 5935 scope.go:122] "RemoveContainer" containerID="8c2e3d02799dd914f0c8a8e4513791276a208976e1a1a7bc48ffc3d73d2cdf3e" Nov 01 20:45:16.398639 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:45:16.398612 5935 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29892765-8fxfg_odh-ai-gateway-infra(8d710b1f-2bab-406e-9e4c-6e39672284d8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892765-8fxfg" podUID="8d710b1f-2bab-406e-9e4c-6e39672284d8" Nov 01 20:45:17.432222 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:17.432184 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892765-8fxfg"] Nov 01 20:45:17.650369 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:17.650342 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892765-8fxfg" Nov 01 20:45:17.780774 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:17.780742 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d710b1f-2bab-406e-9e4c-6e39672284d8-kube-api-access-jccb8\" (UniqueName: \"kubernetes.io/projected/8d710b1f-2bab-406e-9e4c-6e39672284d8-kube-api-access-jccb8\") pod \"8d710b1f-2bab-406e-9e4c-6e39672284d8\" (UID: \"8d710b1f-2bab-406e-9e4c-6e39672284d8\") " Nov 01 20:45:17.780955 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:17.780791 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d710b1f-2bab-406e-9e4c-6e39672284d8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d710b1f-2bab-406e-9e4c-6e39672284d8-maas-api-service-ca\") pod \"8d710b1f-2bab-406e-9e4c-6e39672284d8\" (UID: \"8d710b1f-2bab-406e-9e4c-6e39672284d8\") " Nov 01 20:45:17.781238 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:17.781210 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d710b1f-2bab-406e-9e4c-6e39672284d8-maas-api-service-ca" pod "8d710b1f-2bab-406e-9e4c-6e39672284d8" (UID: "8d710b1f-2bab-406e-9e4c-6e39672284d8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:45:17.782902 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:17.782878 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d710b1f-2bab-406e-9e4c-6e39672284d8-kube-api-access-jccb8" pod "8d710b1f-2bab-406e-9e4c-6e39672284d8" (UID: "8d710b1f-2bab-406e-9e4c-6e39672284d8"). InnerVolumeSpecName "kube-api-access-jccb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:45:17.882011 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:17.881983 5935 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d710b1f-2bab-406e-9e4c-6e39672284d8-maas-api-service-ca\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:45:17.882011 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:17.882007 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jccb8\" (UniqueName: \"kubernetes.io/projected/8d710b1f-2bab-406e-9e4c-6e39672284d8-kube-api-access-jccb8\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:45:18.441157 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:18.441120 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892765-8fxfg" Nov 01 20:45:18.441633 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:18.441122 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892765-8fxfg" event={"ID":"8d710b1f-2bab-406e-9e4c-6e39672284d8","Type":"ContainerDied","Data":"997748c8375844e10781631a741409655011d31b7701da31a79b483c5c01a83f"} Nov 01 20:45:18.441633 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:18.441259 5935 scope.go:122] "RemoveContainer" containerID="8c2e3d02799dd914f0c8a8e4513791276a208976e1a1a7bc48ffc3d73d2cdf3e" Nov 01 20:45:18.458225 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:18.458197 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892765-8fxfg"] Nov 01 20:45:18.463964 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:18.463932 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892765-8fxfg"] Nov 01 20:45:20.366309 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:20.365602 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d710b1f-2bab-406e-9e4c-6e39672284d8" path="/var/lib/kubelet/pods/8d710b1f-2bab-406e-9e4c-6e39672284d8/volumes" Nov 01 20:45:21.776043 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:21.776008 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-724731db-lv9g8"] Nov 01 20:45:21.776586 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:21.776406 5935 cpu_manager.go:414] "containerMap: removing container" podUID="8d710b1f-2bab-406e-9e4c-6e39672284d8" containerName="cleanup" Nov 01 20:45:21.776586 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:21.776418 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d710b1f-2bab-406e-9e4c-6e39672284d8" containerName="cleanup" Nov 01 20:45:21.776586 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:21.776428 5935 cpu_manager.go:414] "containerMap: removing container" podUID="8d710b1f-2bab-406e-9e4c-6e39672284d8" containerName="cleanup" Nov 01 20:45:21.776586 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:21.776433 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d710b1f-2bab-406e-9e4c-6e39672284d8" containerName="cleanup" Nov 01 20:45:21.776586 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:21.776444 5935 cpu_manager.go:414] "containerMap: removing container" podUID="8d710b1f-2bab-406e-9e4c-6e39672284d8" containerName="cleanup" Nov 01 20:45:21.776586 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:21.776449 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d710b1f-2bab-406e-9e4c-6e39672284d8" containerName="cleanup" Nov 01 20:45:21.776586 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:21.776514 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d710b1f-2bab-406e-9e4c-6e39672284d8" containerName="cleanup" Nov 01 20:45:21.776586 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:21.776527 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d710b1f-2bab-406e-9e4c-6e39672284d8" containerName="cleanup" Nov 01 20:45:21.776586 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:21.776532 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d710b1f-2bab-406e-9e4c-6e39672284d8" containerName="cleanup" Nov 01 20:45:21.781053 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:21.781031 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-724731db-lv9g8" Nov 01 20:45:21.785901 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:21.785871 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 01 20:45:21.786163 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:21.786135 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6rcz6\"" Nov 01 20:45:21.787413 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:21.787390 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-724731db-lv9g8"] Nov 01 20:45:21.829482 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:21.829459 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2969417c-c03a-486f-8b51-47d0f2ebf403-maas-api-xks-ca\") pod \"maas-api-revoke-sub-724731db-lv9g8\" (UID: \"2969417c-c03a-486f-8b51-47d0f2ebf403\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-724731db-lv9g8" Nov 01 20:45:21.829607 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:21.829502 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2969417c-c03a-486f-8b51-47d0f2ebf403-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-724731db-lv9g8\" (UID: \"2969417c-c03a-486f-8b51-47d0f2ebf403\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-724731db-lv9g8" Nov 01 20:45:21.829676 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:21.829624 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzq92\" (UniqueName: \"kubernetes.io/projected/2969417c-c03a-486f-8b51-47d0f2ebf403-kube-api-access-mzq92\") pod \"maas-api-revoke-sub-724731db-lv9g8\" (UID: \"2969417c-c03a-486f-8b51-47d0f2ebf403\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-724731db-lv9g8" Nov 01 20:45:21.930478 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:21.930447 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzq92\" (UniqueName: \"kubernetes.io/projected/2969417c-c03a-486f-8b51-47d0f2ebf403-kube-api-access-mzq92\") pod \"maas-api-revoke-sub-724731db-lv9g8\" (UID: \"2969417c-c03a-486f-8b51-47d0f2ebf403\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-724731db-lv9g8" Nov 01 20:45:21.930621 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:21.930530 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2969417c-c03a-486f-8b51-47d0f2ebf403-maas-api-xks-ca\") pod \"maas-api-revoke-sub-724731db-lv9g8\" (UID: \"2969417c-c03a-486f-8b51-47d0f2ebf403\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-724731db-lv9g8" Nov 01 20:45:21.930621 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:21.930565 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2969417c-c03a-486f-8b51-47d0f2ebf403-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-724731db-lv9g8\" (UID: \"2969417c-c03a-486f-8b51-47d0f2ebf403\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-724731db-lv9g8" Nov 01 20:45:21.931149 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:21.931127 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2969417c-c03a-486f-8b51-47d0f2ebf403-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-724731db-lv9g8\" (UID: \"2969417c-c03a-486f-8b51-47d0f2ebf403\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-724731db-lv9g8" Nov 01 20:45:21.932852 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:21.932821 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2969417c-c03a-486f-8b51-47d0f2ebf403-maas-api-xks-ca\") pod \"maas-api-revoke-sub-724731db-lv9g8\" (UID: \"2969417c-c03a-486f-8b51-47d0f2ebf403\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-724731db-lv9g8" Nov 01 20:45:21.938033 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:21.938009 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzq92\" (UniqueName: \"kubernetes.io/projected/2969417c-c03a-486f-8b51-47d0f2ebf403-kube-api-access-mzq92\") pod \"maas-api-revoke-sub-724731db-lv9g8\" (UID: \"2969417c-c03a-486f-8b51-47d0f2ebf403\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-724731db-lv9g8" Nov 01 20:45:22.099748 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:22.099696 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-724731db-lv9g8" Nov 01 20:45:22.371841 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:22.371752 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-724731db-lv9g8"] Nov 01 20:45:22.376128 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:45:22.376099 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2969417c_c03a_486f_8b51_47d0f2ebf403.slice/crio-11666e2d7bb43091f03d2d08ffe96ef00b3eb5093ccdafff07294dd0979cff0c WatchSource:0}: Error finding container 11666e2d7bb43091f03d2d08ffe96ef00b3eb5093ccdafff07294dd0979cff0c: Status 404 returned error can't find the container with id 11666e2d7bb43091f03d2d08ffe96ef00b3eb5093ccdafff07294dd0979cff0c Nov 01 20:45:22.502670 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:22.502629 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-724731db-lv9g8" event={"ID":"2969417c-c03a-486f-8b51-47d0f2ebf403","Type":"ContainerStarted","Data":"11666e2d7bb43091f03d2d08ffe96ef00b3eb5093ccdafff07294dd0979cff0c"} Nov 01 20:45:23.515314 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:23.515169 5935 generic.go:356] "Generic (PLEG): container finished" podID="2969417c-c03a-486f-8b51-47d0f2ebf403" containerID="1377f42472c1605728e3ecfd8f14ed7de3170947447bab51f8a1d31bcdd9e634" exitCode=0 Nov 01 20:45:23.516249 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:23.515888 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-724731db-lv9g8" event={"ID":"2969417c-c03a-486f-8b51-47d0f2ebf403","Type":"ContainerDied","Data":"1377f42472c1605728e3ecfd8f14ed7de3170947447bab51f8a1d31bcdd9e634"} Nov 01 20:45:24.785102 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:24.785044 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-724731db-lv9g8" Nov 01 20:45:24.900683 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:24.900650 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2969417c-c03a-486f-8b51-47d0f2ebf403-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2969417c-c03a-486f-8b51-47d0f2ebf403-maas-api-openshift-ca\") pod \"2969417c-c03a-486f-8b51-47d0f2ebf403\" (UID: \"2969417c-c03a-486f-8b51-47d0f2ebf403\") " Nov 01 20:45:24.900845 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:24.900707 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2969417c-c03a-486f-8b51-47d0f2ebf403-kube-api-access-mzq92\" (UniqueName: \"kubernetes.io/projected/2969417c-c03a-486f-8b51-47d0f2ebf403-kube-api-access-mzq92\") pod \"2969417c-c03a-486f-8b51-47d0f2ebf403\" (UID: \"2969417c-c03a-486f-8b51-47d0f2ebf403\") " Nov 01 20:45:24.900845 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:24.900753 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2969417c-c03a-486f-8b51-47d0f2ebf403-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2969417c-c03a-486f-8b51-47d0f2ebf403-maas-api-xks-ca\") pod \"2969417c-c03a-486f-8b51-47d0f2ebf403\" (UID: \"2969417c-c03a-486f-8b51-47d0f2ebf403\") " Nov 01 20:45:24.901012 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:24.900989 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2969417c-c03a-486f-8b51-47d0f2ebf403-maas-api-openshift-ca" pod "2969417c-c03a-486f-8b51-47d0f2ebf403" (UID: "2969417c-c03a-486f-8b51-47d0f2ebf403"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:45:24.901087 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:24.901067 5935 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2969417c-c03a-486f-8b51-47d0f2ebf403-maas-api-openshift-ca\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:45:24.902758 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:24.902731 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2969417c-c03a-486f-8b51-47d0f2ebf403-kube-api-access-mzq92" pod "2969417c-c03a-486f-8b51-47d0f2ebf403" (UID: "2969417c-c03a-486f-8b51-47d0f2ebf403"). InnerVolumeSpecName "kube-api-access-mzq92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:45:24.902841 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:24.902821 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2969417c-c03a-486f-8b51-47d0f2ebf403-maas-api-xks-ca" pod "2969417c-c03a-486f-8b51-47d0f2ebf403" (UID: "2969417c-c03a-486f-8b51-47d0f2ebf403"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:45:25.001490 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:25.001465 5935 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2969417c-c03a-486f-8b51-47d0f2ebf403-maas-api-xks-ca\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:45:25.001490 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:25.001486 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzq92\" (UniqueName: \"kubernetes.io/projected/2969417c-c03a-486f-8b51-47d0f2ebf403-kube-api-access-mzq92\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:45:25.528467 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:25.528435 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-724731db-lv9g8" Nov 01 20:45:25.528648 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:25.528477 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-724731db-lv9g8" event={"ID":"2969417c-c03a-486f-8b51-47d0f2ebf403","Type":"ContainerDied","Data":"11666e2d7bb43091f03d2d08ffe96ef00b3eb5093ccdafff07294dd0979cff0c"} Nov 01 20:45:25.528648 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:45:25.528520 5935 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11666e2d7bb43091f03d2d08ffe96ef00b3eb5093ccdafff07294dd0979cff0c" Nov 01 20:45:35.845658 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:45:35.845610 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:46:05.842179 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:46:05.842138 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:46:35.843636 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:46:35.843598 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:46:41.705469 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:46:41.705423 5935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-ljtcq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.22:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 20:46:41.709874 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:46:41.706021 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-ljtcq" podUID="03014616-4520-48bb-a548-66485577e26c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.22:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:47:05.844889 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:47:05.844855 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:47:35.842940 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:47:35.842852 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:47:39.656436 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:47:39.656398 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a97e34-6fbc9fc65-hz65n"] Nov 01 20:47:39.656851 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:47:39.656760 5935 cpu_manager.go:414] "containerMap: removing container" podUID="2969417c-c03a-486f-8b51-47d0f2ebf403" containerName="revoke-keys" Nov 01 20:47:39.656851 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:47:39.656771 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2969417c-c03a-486f-8b51-47d0f2ebf403" containerName="revoke-keys" Nov 01 20:47:39.656851 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:47:39.656846 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="2969417c-c03a-486f-8b51-47d0f2ebf403" containerName="revoke-keys" Nov 01 20:47:39.665128 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:47:39.665099 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a97e34-6fbc9fc65-hz65n" Nov 01 20:47:39.668577 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:47:39.668552 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a97e34-6fbc9fc65-hz65n"] Nov 01 20:47:39.675365 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:47:39.674343 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-a97e34-dockercfg-zl2f5\"" Nov 01 20:47:39.675365 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:47:39.674797 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a97e34\"" Nov 01 20:47:39.685126 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:47:39.685097 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a97e34-54c9b6ff9f-9cjv2"] Nov 01 20:47:39.692208 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:47:39.692186 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a97e34-54c9b6ff9f-9cjv2" Nov 01 20:47:39.696993 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:47:39.696971 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a97e34-54c9b6ff9f-9cjv2"] Nov 01 20:47:39.772001 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:47:39.771959 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx8qd\" (UniqueName: \"kubernetes.io/projected/d19a1249-202e-496f-b3e3-74f793f056a9-kube-api-access-lx8qd\") pod \"payload-processing-e2e-ipp-b-a97e34-6fbc9fc65-hz65n\" (UID: \"d19a1249-202e-496f-b3e3-74f793f056a9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a97e34-6fbc9fc65-hz65n" Nov 01 20:47:39.772408 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:47:39.772042 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d19a1249-202e-496f-b3e3-74f793f056a9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a97e34-6fbc9fc65-hz65n\" (UID: \"d19a1249-202e-496f-b3e3-74f793f056a9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a97e34-6fbc9fc65-hz65n" Nov 01 20:47:39.772408 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:47:39.772162 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/422eb76c-3274-4c49-937d-2cfc5b92f813-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a97e34-54c9b6ff9f-9cjv2\" (UID: \"422eb76c-3274-4c49-937d-2cfc5b92f813\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a97e34-54c9b6ff9f-9cjv2" Nov 01 20:47:39.772408 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:47:39.772401 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjqpr\" (UniqueName: \"kubernetes.io/projected/422eb76c-3274-4c49-937d-2cfc5b92f813-kube-api-access-pjqpr\") pod \"payload-pre-processing-e2e-ipp-b-a97e34-54c9b6ff9f-9cjv2\" (UID: \"422eb76c-3274-4c49-937d-2cfc5b92f813\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a97e34-54c9b6ff9f-9cjv2" Nov 01 20:47:39.873853 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:47:39.873825 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d19a1249-202e-496f-b3e3-74f793f056a9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a97e34-6fbc9fc65-hz65n\" (UID: \"d19a1249-202e-496f-b3e3-74f793f056a9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a97e34-6fbc9fc65-hz65n" Nov 01 20:47:39.874030 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:47:39.873918 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/422eb76c-3274-4c49-937d-2cfc5b92f813-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a97e34-54c9b6ff9f-9cjv2\" (UID: \"422eb76c-3274-4c49-937d-2cfc5b92f813\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a97e34-54c9b6ff9f-9cjv2" Nov 01 20:47:39.874030 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:47:39.873997 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjqpr\" (UniqueName: \"kubernetes.io/projected/422eb76c-3274-4c49-937d-2cfc5b92f813-kube-api-access-pjqpr\") pod \"payload-pre-processing-e2e-ipp-b-a97e34-54c9b6ff9f-9cjv2\" (UID: \"422eb76c-3274-4c49-937d-2cfc5b92f813\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a97e34-54c9b6ff9f-9cjv2" Nov 01 20:47:39.874030 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:47:39.874024 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx8qd\" (UniqueName: \"kubernetes.io/projected/d19a1249-202e-496f-b3e3-74f793f056a9-kube-api-access-lx8qd\") pod \"payload-processing-e2e-ipp-b-a97e34-6fbc9fc65-hz65n\" (UID: \"d19a1249-202e-496f-b3e3-74f793f056a9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a97e34-6fbc9fc65-hz65n" Nov 01 20:47:39.874725 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:47:39.874702 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d19a1249-202e-496f-b3e3-74f793f056a9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a97e34-6fbc9fc65-hz65n\" (UID: \"d19a1249-202e-496f-b3e3-74f793f056a9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a97e34-6fbc9fc65-hz65n" Nov 01 20:47:39.874839 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:47:39.874710 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/422eb76c-3274-4c49-937d-2cfc5b92f813-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a97e34-54c9b6ff9f-9cjv2\" (UID: \"422eb76c-3274-4c49-937d-2cfc5b92f813\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a97e34-54c9b6ff9f-9cjv2" Nov 01 20:47:39.882166 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:47:39.882137 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjqpr\" (UniqueName: \"kubernetes.io/projected/422eb76c-3274-4c49-937d-2cfc5b92f813-kube-api-access-pjqpr\") pod \"payload-pre-processing-e2e-ipp-b-a97e34-54c9b6ff9f-9cjv2\" (UID: \"422eb76c-3274-4c49-937d-2cfc5b92f813\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a97e34-54c9b6ff9f-9cjv2" Nov 01 20:47:39.882266 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:47:39.882190 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx8qd\" (UniqueName: \"kubernetes.io/projected/d19a1249-202e-496f-b3e3-74f793f056a9-kube-api-access-lx8qd\") pod \"payload-processing-e2e-ipp-b-a97e34-6fbc9fc65-hz65n\" (UID: \"d19a1249-202e-496f-b3e3-74f793f056a9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a97e34-6fbc9fc65-hz65n" Nov 01 20:47:39.983064 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:47:39.983032 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a97e34-6fbc9fc65-hz65n" Nov 01 20:47:40.009351 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:47:40.009323 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a97e34-54c9b6ff9f-9cjv2" Nov 01 20:47:40.255175 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:47:40.255149 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a97e34-6fbc9fc65-hz65n"] Nov 01 20:47:40.257344 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:47:40.257314 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd19a1249_202e_496f_b3e3_74f793f056a9.slice/crio-cd5c1d4148d1c60903807f77bde56629548db8c56357c3ee119c81ef81af1b0c WatchSource:0}: Error finding container cd5c1d4148d1c60903807f77bde56629548db8c56357c3ee119c81ef81af1b0c: Status 404 returned error can't find the container with id cd5c1d4148d1c60903807f77bde56629548db8c56357c3ee119c81ef81af1b0c Nov 01 20:47:40.295924 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:47:40.295903 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a97e34-54c9b6ff9f-9cjv2"] Nov 01 20:47:40.300446 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:47:40.300422 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod422eb76c_3274_4c49_937d_2cfc5b92f813.slice/crio-0ca5d6f2c096b4afd8aaa712fecae9de0c9d184edf08b4a8ef377b396984f530 WatchSource:0}: Error finding container 0ca5d6f2c096b4afd8aaa712fecae9de0c9d184edf08b4a8ef377b396984f530: Status 404 returned error can't find the container with id 0ca5d6f2c096b4afd8aaa712fecae9de0c9d184edf08b4a8ef377b396984f530 Nov 01 20:47:40.308622 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:47:40.308585 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a97e34-6fbc9fc65-hz65n" event={"ID":"d19a1249-202e-496f-b3e3-74f793f056a9","Type":"ContainerStarted","Data":"cd5c1d4148d1c60903807f77bde56629548db8c56357c3ee119c81ef81af1b0c"} Nov 01 20:47:40.309886 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:47:40.309865 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a97e34-54c9b6ff9f-9cjv2" event={"ID":"422eb76c-3274-4c49-937d-2cfc5b92f813","Type":"ContainerStarted","Data":"0ca5d6f2c096b4afd8aaa712fecae9de0c9d184edf08b4a8ef377b396984f530"} Nov 01 20:47:41.316259 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:47:41.316219 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a97e34-54c9b6ff9f-9cjv2" event={"ID":"422eb76c-3274-4c49-937d-2cfc5b92f813","Type":"ContainerStarted","Data":"902db1530ae0e854f7e5810515221d788554866b31cb63b68a82154cb4ed8634"} Nov 01 20:47:41.316716 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:47:41.316305 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a97e34-54c9b6ff9f-9cjv2" Nov 01 20:47:41.318002 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:47:41.317977 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a97e34-6fbc9fc65-hz65n" event={"ID":"d19a1249-202e-496f-b3e3-74f793f056a9","Type":"ContainerStarted","Data":"1ed14fdee07948864b10ff16482bfee05516a9a9f4ef42890a9dc699168d9dc8"} Nov 01 20:47:41.318165 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:47:41.318108 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a97e34-6fbc9fc65-hz65n" Nov 01 20:47:41.354705 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:47:41.354664 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a97e34-54c9b6ff9f-9cjv2" podStartSLOduration=2.354652463 podStartE2EDuration="2.354652463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:47:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:47:41.334246957 +0000 UTC m=+2907.107407469" watchObservedRunningTime="2026-11-01 20:47:41.354652463 +0000 UTC m=+2907.127812955" Nov 01 20:47:41.356032 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:47:41.355995 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-a97e34-6fbc9fc65-hz65n" podStartSLOduration=2.355984393 podStartE2EDuration="2.355984393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:47:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:47:41.35478436 +0000 UTC m=+2907.127944856" watchObservedRunningTime="2026-11-01 20:47:41.355984393 +0000 UTC m=+2907.129144905" Nov 01 20:47:52.326149 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:47:52.326123 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a97e34-6fbc9fc65-hz65n" Nov 01 20:47:52.326651 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:47:52.326199 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a97e34-54c9b6ff9f-9cjv2" Nov 01 20:47:53.375925 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:47:53.375891 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a97e34-6fbc9fc65-hz65n"] Nov 01 20:47:53.389696 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:47:53.389670 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a97e34-54c9b6ff9f-9cjv2"] Nov 01 20:47:58.376383 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:47:58.376345 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-a97e34-6fbc9fc65-hz65n" podUID="d19a1249-202e-496f-b3e3-74f793f056a9" containerName="payload-processing" containerID="cri-o://1ed14fdee07948864b10ff16482bfee05516a9a9f4ef42890a9dc699168d9dc8" gracePeriod=25 Nov 01 20:47:58.390321 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:47:58.390247 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a97e34-54c9b6ff9f-9cjv2" podUID="422eb76c-3274-4c49-937d-2cfc5b92f813" containerName="payload-pre-processing" containerID="cri-o://902db1530ae0e854f7e5810515221d788554866b31cb63b68a82154cb4ed8634" gracePeriod=25 Nov 01 20:47:58.624777 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:47:58.624750 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a97e34-6fbc9fc65-hz65n" Nov 01 20:47:58.656263 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:47:58.656238 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a97e34-54c9b6ff9f-9cjv2" Nov 01 20:47:58.757597 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:47:58.757557 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/422eb76c-3274-4c49-937d-2cfc5b92f813-kube-api-access-pjqpr\" (UniqueName: \"kubernetes.io/projected/422eb76c-3274-4c49-937d-2cfc5b92f813-kube-api-access-pjqpr\") pod \"422eb76c-3274-4c49-937d-2cfc5b92f813\" (UID: \"422eb76c-3274-4c49-937d-2cfc5b92f813\") " Nov 01 20:47:58.757768 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:47:58.757614 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d19a1249-202e-496f-b3e3-74f793f056a9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d19a1249-202e-496f-b3e3-74f793f056a9-plugins-config-volume\") pod \"d19a1249-202e-496f-b3e3-74f793f056a9\" (UID: \"d19a1249-202e-496f-b3e3-74f793f056a9\") " Nov 01 20:47:58.757768 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:47:58.757691 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d19a1249-202e-496f-b3e3-74f793f056a9-kube-api-access-lx8qd\" (UniqueName: \"kubernetes.io/projected/d19a1249-202e-496f-b3e3-74f793f056a9-kube-api-access-lx8qd\") pod \"d19a1249-202e-496f-b3e3-74f793f056a9\" (UID: \"d19a1249-202e-496f-b3e3-74f793f056a9\") " Nov 01 20:47:58.757885 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:47:58.757765 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/422eb76c-3274-4c49-937d-2cfc5b92f813-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/422eb76c-3274-4c49-937d-2cfc5b92f813-plugins-config-volume\") pod \"422eb76c-3274-4c49-937d-2cfc5b92f813\" (UID: \"422eb76c-3274-4c49-937d-2cfc5b92f813\") " Nov 01 20:47:58.758086 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:47:58.758060 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d19a1249-202e-496f-b3e3-74f793f056a9-plugins-config-volume" pod "d19a1249-202e-496f-b3e3-74f793f056a9" (UID: "d19a1249-202e-496f-b3e3-74f793f056a9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:47:58.758176 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:47:58.758143 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/422eb76c-3274-4c49-937d-2cfc5b92f813-plugins-config-volume" pod "422eb76c-3274-4c49-937d-2cfc5b92f813" (UID: "422eb76c-3274-4c49-937d-2cfc5b92f813"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:47:58.759546 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:47:58.759526 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/422eb76c-3274-4c49-937d-2cfc5b92f813-kube-api-access-pjqpr" pod "422eb76c-3274-4c49-937d-2cfc5b92f813" (UID: "422eb76c-3274-4c49-937d-2cfc5b92f813"). InnerVolumeSpecName "kube-api-access-pjqpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:47:58.759668 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:47:58.759648 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d19a1249-202e-496f-b3e3-74f793f056a9-kube-api-access-lx8qd" pod "d19a1249-202e-496f-b3e3-74f793f056a9" (UID: "d19a1249-202e-496f-b3e3-74f793f056a9"). InnerVolumeSpecName "kube-api-access-lx8qd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:47:58.859087 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:47:58.859043 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lx8qd\" (UniqueName: \"kubernetes.io/projected/d19a1249-202e-496f-b3e3-74f793f056a9-kube-api-access-lx8qd\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:47:58.859087 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:47:58.859082 5935 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/422eb76c-3274-4c49-937d-2cfc5b92f813-plugins-config-volume\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:47:58.859087 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:47:58.859093 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjqpr\" (UniqueName: \"kubernetes.io/projected/422eb76c-3274-4c49-937d-2cfc5b92f813-kube-api-access-pjqpr\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:47:58.859087 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:47:58.859104 5935 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d19a1249-202e-496f-b3e3-74f793f056a9-plugins-config-volume\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:47:59.408949 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:47:59.408915 5935 generic.go:356] "Generic (PLEG): container finished" podID="422eb76c-3274-4c49-937d-2cfc5b92f813" containerID="902db1530ae0e854f7e5810515221d788554866b31cb63b68a82154cb4ed8634" exitCode=0 Nov 01 20:47:59.409328 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:47:59.408969 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a97e34-54c9b6ff9f-9cjv2" Nov 01 20:47:59.409328 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:47:59.408998 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a97e34-54c9b6ff9f-9cjv2" event={"ID":"422eb76c-3274-4c49-937d-2cfc5b92f813","Type":"ContainerDied","Data":"902db1530ae0e854f7e5810515221d788554866b31cb63b68a82154cb4ed8634"} Nov 01 20:47:59.409328 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:47:59.409047 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a97e34-54c9b6ff9f-9cjv2" event={"ID":"422eb76c-3274-4c49-937d-2cfc5b92f813","Type":"ContainerDied","Data":"0ca5d6f2c096b4afd8aaa712fecae9de0c9d184edf08b4a8ef377b396984f530"} Nov 01 20:47:59.409328 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:47:59.409068 5935 scope.go:122] "RemoveContainer" containerID="902db1530ae0e854f7e5810515221d788554866b31cb63b68a82154cb4ed8634" Nov 01 20:47:59.410718 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:47:59.410692 5935 generic.go:356] "Generic (PLEG): container finished" podID="d19a1249-202e-496f-b3e3-74f793f056a9" containerID="1ed14fdee07948864b10ff16482bfee05516a9a9f4ef42890a9dc699168d9dc8" exitCode=0 Nov 01 20:47:59.410816 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:47:59.410727 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a97e34-6fbc9fc65-hz65n" event={"ID":"d19a1249-202e-496f-b3e3-74f793f056a9","Type":"ContainerDied","Data":"1ed14fdee07948864b10ff16482bfee05516a9a9f4ef42890a9dc699168d9dc8"} Nov 01 20:47:59.410816 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:47:59.410758 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a97e34-6fbc9fc65-hz65n" event={"ID":"d19a1249-202e-496f-b3e3-74f793f056a9","Type":"ContainerDied","Data":"cd5c1d4148d1c60903807f77bde56629548db8c56357c3ee119c81ef81af1b0c"} Nov 01 20:47:59.410816 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:47:59.410731 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a97e34-6fbc9fc65-hz65n" Nov 01 20:47:59.419768 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:47:59.419754 5935 scope.go:122] "RemoveContainer" containerID="902db1530ae0e854f7e5810515221d788554866b31cb63b68a82154cb4ed8634" Nov 01 20:47:59.420041 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:47:59.420021 5935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"902db1530ae0e854f7e5810515221d788554866b31cb63b68a82154cb4ed8634\": container with ID starting with 902db1530ae0e854f7e5810515221d788554866b31cb63b68a82154cb4ed8634 not found: ID does not exist" containerID="902db1530ae0e854f7e5810515221d788554866b31cb63b68a82154cb4ed8634" Nov 01 20:47:59.420123 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:47:59.420046 5935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"902db1530ae0e854f7e5810515221d788554866b31cb63b68a82154cb4ed8634"} err="failed to get container status \"902db1530ae0e854f7e5810515221d788554866b31cb63b68a82154cb4ed8634\": rpc error: code = NotFound desc = could not find container \"902db1530ae0e854f7e5810515221d788554866b31cb63b68a82154cb4ed8634\": container with ID starting with 902db1530ae0e854f7e5810515221d788554866b31cb63b68a82154cb4ed8634 not found: ID does not exist" Nov 01 20:47:59.420123 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:47:59.420063 5935 scope.go:122] "RemoveContainer" containerID="1ed14fdee07948864b10ff16482bfee05516a9a9f4ef42890a9dc699168d9dc8" Nov 01 20:47:59.428000 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:47:59.427984 5935 scope.go:122] "RemoveContainer" containerID="1ed14fdee07948864b10ff16482bfee05516a9a9f4ef42890a9dc699168d9dc8" Nov 01 20:47:59.428274 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:47:59.428240 5935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ed14fdee07948864b10ff16482bfee05516a9a9f4ef42890a9dc699168d9dc8\": container with ID starting with 1ed14fdee07948864b10ff16482bfee05516a9a9f4ef42890a9dc699168d9dc8 not found: ID does not exist" containerID="1ed14fdee07948864b10ff16482bfee05516a9a9f4ef42890a9dc699168d9dc8" Nov 01 20:47:59.428377 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:47:59.428297 5935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ed14fdee07948864b10ff16482bfee05516a9a9f4ef42890a9dc699168d9dc8"} err="failed to get container status \"1ed14fdee07948864b10ff16482bfee05516a9a9f4ef42890a9dc699168d9dc8\": rpc error: code = NotFound desc = could not find container \"1ed14fdee07948864b10ff16482bfee05516a9a9f4ef42890a9dc699168d9dc8\": container with ID starting with 1ed14fdee07948864b10ff16482bfee05516a9a9f4ef42890a9dc699168d9dc8 not found: ID does not exist" Nov 01 20:47:59.431209 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:47:59.431189 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a97e34-54c9b6ff9f-9cjv2"] Nov 01 20:47:59.432724 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:47:59.432702 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a97e34-54c9b6ff9f-9cjv2"] Nov 01 20:47:59.441749 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:47:59.441731 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a97e34-6fbc9fc65-hz65n"] Nov 01 20:47:59.445247 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:47:59.445229 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a97e34-6fbc9fc65-hz65n"] Nov 01 20:48:00.358737 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:48:00.358704 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="422eb76c-3274-4c49-937d-2cfc5b92f813" path="/var/lib/kubelet/pods/422eb76c-3274-4c49-937d-2cfc5b92f813/volumes" Nov 01 20:48:00.359197 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:48:00.359178 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d19a1249-202e-496f-b3e3-74f793f056a9" path="/var/lib/kubelet/pods/d19a1249-202e-496f-b3e3-74f793f056a9/volumes" Nov 01 20:48:05.842648 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:48:05.842603 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:48:16.624946 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:48:16.624909 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-c268d1/ipp-route-c268d1-kserve-d56f88c6f-wjs96"] Nov 01 20:48:16.625489 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:48:16.625372 5935 cpu_manager.go:414] "containerMap: removing container" podUID="422eb76c-3274-4c49-937d-2cfc5b92f813" containerName="payload-pre-processing" Nov 01 20:48:16.625489 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:48:16.625392 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="422eb76c-3274-4c49-937d-2cfc5b92f813" containerName="payload-pre-processing" Nov 01 20:48:16.625489 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:48:16.625452 5935 cpu_manager.go:414] "containerMap: removing container" podUID="d19a1249-202e-496f-b3e3-74f793f056a9" containerName="payload-processing" Nov 01 20:48:16.625489 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:48:16.625476 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d19a1249-202e-496f-b3e3-74f793f056a9" containerName="payload-processing" Nov 01 20:48:16.625783 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:48:16.625580 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="d19a1249-202e-496f-b3e3-74f793f056a9" containerName="payload-processing" Nov 01 20:48:16.625783 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:48:16.625597 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="422eb76c-3274-4c49-937d-2cfc5b92f813" containerName="payload-pre-processing" Nov 01 20:48:16.631406 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:48:16.631382 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-c268d1/ipp-route-c268d1-kserve-d56f88c6f-wjs96" Nov 01 20:48:16.639086 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:48:16.639050 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-c268d1/ipp-route-c268d1-kserve-d56f88c6f-wjs96"] Nov 01 20:48:16.640711 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:48:16.640688 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-c268d1\"/\"default-dockercfg-vs4nn\"" Nov 01 20:48:16.640826 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:48:16.640800 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-c268d1\"/\"ipp-route-c268d1-kserve-self-signed-certs\"" Nov 01 20:48:16.712047 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:48:16.712013 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf19ba79-a4e7-42bf-841e-47a105121445-tmp-dir\") pod \"ipp-route-c268d1-kserve-d56f88c6f-wjs96\" (UID: \"cf19ba79-a4e7-42bf-841e-47a105121445\") " pod="ai-tenant-e2e-ipp-a-c268d1/ipp-route-c268d1-kserve-d56f88c6f-wjs96" Nov 01 20:48:16.712047 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:48:16.712050 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf19ba79-a4e7-42bf-841e-47a105121445-dshm\") pod \"ipp-route-c268d1-kserve-d56f88c6f-wjs96\" (UID: \"cf19ba79-a4e7-42bf-841e-47a105121445\") " pod="ai-tenant-e2e-ipp-a-c268d1/ipp-route-c268d1-kserve-d56f88c6f-wjs96" Nov 01 20:48:16.712332 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:48:16.712129 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf19ba79-a4e7-42bf-841e-47a105121445-tls-certs\") pod \"ipp-route-c268d1-kserve-d56f88c6f-wjs96\" (UID: \"cf19ba79-a4e7-42bf-841e-47a105121445\") " pod="ai-tenant-e2e-ipp-a-c268d1/ipp-route-c268d1-kserve-d56f88c6f-wjs96" Nov 01 20:48:16.712332 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:48:16.712166 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmx8w\" (UniqueName: \"kubernetes.io/projected/cf19ba79-a4e7-42bf-841e-47a105121445-kube-api-access-wmx8w\") pod \"ipp-route-c268d1-kserve-d56f88c6f-wjs96\" (UID: \"cf19ba79-a4e7-42bf-841e-47a105121445\") " pod="ai-tenant-e2e-ipp-a-c268d1/ipp-route-c268d1-kserve-d56f88c6f-wjs96" Nov 01 20:48:16.712332 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:48:16.712217 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf19ba79-a4e7-42bf-841e-47a105121445-model-cache\") pod \"ipp-route-c268d1-kserve-d56f88c6f-wjs96\" (UID: \"cf19ba79-a4e7-42bf-841e-47a105121445\") " pod="ai-tenant-e2e-ipp-a-c268d1/ipp-route-c268d1-kserve-d56f88c6f-wjs96" Nov 01 20:48:16.712513 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:48:16.712339 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf19ba79-a4e7-42bf-841e-47a105121445-home\") pod \"ipp-route-c268d1-kserve-d56f88c6f-wjs96\" (UID: \"cf19ba79-a4e7-42bf-841e-47a105121445\") " pod="ai-tenant-e2e-ipp-a-c268d1/ipp-route-c268d1-kserve-d56f88c6f-wjs96" Nov 01 20:48:16.716840 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:48:16.716817 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-c268d1\"/\"kube-root-ca.crt\"" Nov 01 20:48:16.716982 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:48:16.716916 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-c268d1\"/\"openshift-service-ca.crt\"" Nov 01 20:48:16.813876 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:48:16.813840 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf19ba79-a4e7-42bf-841e-47a105121445-model-cache\") pod \"ipp-route-c268d1-kserve-d56f88c6f-wjs96\" (UID: \"cf19ba79-a4e7-42bf-841e-47a105121445\") " pod="ai-tenant-e2e-ipp-a-c268d1/ipp-route-c268d1-kserve-d56f88c6f-wjs96" Nov 01 20:48:16.814048 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:48:16.813904 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf19ba79-a4e7-42bf-841e-47a105121445-home\") pod \"ipp-route-c268d1-kserve-d56f88c6f-wjs96\" (UID: \"cf19ba79-a4e7-42bf-841e-47a105121445\") " pod="ai-tenant-e2e-ipp-a-c268d1/ipp-route-c268d1-kserve-d56f88c6f-wjs96" Nov 01 20:48:16.814048 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:48:16.813947 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf19ba79-a4e7-42bf-841e-47a105121445-tmp-dir\") pod \"ipp-route-c268d1-kserve-d56f88c6f-wjs96\" (UID: \"cf19ba79-a4e7-42bf-841e-47a105121445\") " pod="ai-tenant-e2e-ipp-a-c268d1/ipp-route-c268d1-kserve-d56f88c6f-wjs96" Nov 01 20:48:16.814048 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:48:16.813963 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf19ba79-a4e7-42bf-841e-47a105121445-dshm\") pod \"ipp-route-c268d1-kserve-d56f88c6f-wjs96\" (UID: \"cf19ba79-a4e7-42bf-841e-47a105121445\") " pod="ai-tenant-e2e-ipp-a-c268d1/ipp-route-c268d1-kserve-d56f88c6f-wjs96" Nov 01 20:48:16.814217 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:48:16.814095 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf19ba79-a4e7-42bf-841e-47a105121445-tls-certs\") pod \"ipp-route-c268d1-kserve-d56f88c6f-wjs96\" (UID: \"cf19ba79-a4e7-42bf-841e-47a105121445\") " pod="ai-tenant-e2e-ipp-a-c268d1/ipp-route-c268d1-kserve-d56f88c6f-wjs96" Nov 01 20:48:16.814217 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:48:16.814144 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmx8w\" (UniqueName: \"kubernetes.io/projected/cf19ba79-a4e7-42bf-841e-47a105121445-kube-api-access-wmx8w\") pod \"ipp-route-c268d1-kserve-d56f88c6f-wjs96\" (UID: \"cf19ba79-a4e7-42bf-841e-47a105121445\") " pod="ai-tenant-e2e-ipp-a-c268d1/ipp-route-c268d1-kserve-d56f88c6f-wjs96" Nov 01 20:48:16.814340 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:48:16.814228 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf19ba79-a4e7-42bf-841e-47a105121445-model-cache\") pod \"ipp-route-c268d1-kserve-d56f88c6f-wjs96\" (UID: \"cf19ba79-a4e7-42bf-841e-47a105121445\") " pod="ai-tenant-e2e-ipp-a-c268d1/ipp-route-c268d1-kserve-d56f88c6f-wjs96" Nov 01 20:48:16.814475 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:48:16.814409 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf19ba79-a4e7-42bf-841e-47a105121445-home\") pod \"ipp-route-c268d1-kserve-d56f88c6f-wjs96\" (UID: \"cf19ba79-a4e7-42bf-841e-47a105121445\") " pod="ai-tenant-e2e-ipp-a-c268d1/ipp-route-c268d1-kserve-d56f88c6f-wjs96" Nov 01 20:48:16.814475 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:48:16.814429 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf19ba79-a4e7-42bf-841e-47a105121445-tmp-dir\") pod \"ipp-route-c268d1-kserve-d56f88c6f-wjs96\" (UID: \"cf19ba79-a4e7-42bf-841e-47a105121445\") " pod="ai-tenant-e2e-ipp-a-c268d1/ipp-route-c268d1-kserve-d56f88c6f-wjs96" Nov 01 20:48:16.816172 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:48:16.816151 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf19ba79-a4e7-42bf-841e-47a105121445-dshm\") pod \"ipp-route-c268d1-kserve-d56f88c6f-wjs96\" (UID: \"cf19ba79-a4e7-42bf-841e-47a105121445\") " pod="ai-tenant-e2e-ipp-a-c268d1/ipp-route-c268d1-kserve-d56f88c6f-wjs96" Nov 01 20:48:16.816670 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:48:16.816648 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf19ba79-a4e7-42bf-841e-47a105121445-tls-certs\") pod \"ipp-route-c268d1-kserve-d56f88c6f-wjs96\" (UID: \"cf19ba79-a4e7-42bf-841e-47a105121445\") " pod="ai-tenant-e2e-ipp-a-c268d1/ipp-route-c268d1-kserve-d56f88c6f-wjs96" Nov 01 20:48:16.821969 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:48:16.821946 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmx8w\" (UniqueName: \"kubernetes.io/projected/cf19ba79-a4e7-42bf-841e-47a105121445-kube-api-access-wmx8w\") pod \"ipp-route-c268d1-kserve-d56f88c6f-wjs96\" (UID: \"cf19ba79-a4e7-42bf-841e-47a105121445\") " pod="ai-tenant-e2e-ipp-a-c268d1/ipp-route-c268d1-kserve-d56f88c6f-wjs96" Nov 01 20:48:16.951164 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:48:16.951138 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-c268d1/ipp-route-c268d1-kserve-d56f88c6f-wjs96" Nov 01 20:48:17.225384 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:48:17.225300 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-c268d1/ipp-route-c268d1-kserve-d56f88c6f-wjs96"] Nov 01 20:48:17.227512 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:48:17.227481 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcf19ba79_a4e7_42bf_841e_47a105121445.slice/crio-47c86c70c314b9e32afb3b41e560917b270f2d151cf9a5f7c2439fde27aa0914 WatchSource:0}: Error finding container 47c86c70c314b9e32afb3b41e560917b270f2d151cf9a5f7c2439fde27aa0914: Status 404 returned error can't find the container with id 47c86c70c314b9e32afb3b41e560917b270f2d151cf9a5f7c2439fde27aa0914 Nov 01 20:48:17.512352 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:48:17.511573 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-c268d1/ipp-route-c268d1-kserve-d56f88c6f-wjs96" event={"ID":"cf19ba79-a4e7-42bf-841e-47a105121445","Type":"ContainerStarted","Data":"aa0b8d6b45da54e33112a528d928d25f32f8b1232a6d9711b360d464b7a7cfcd"} Nov 01 20:48:17.512352 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:48:17.511628 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-c268d1/ipp-route-c268d1-kserve-d56f88c6f-wjs96" event={"ID":"cf19ba79-a4e7-42bf-841e-47a105121445","Type":"ContainerStarted","Data":"47c86c70c314b9e32afb3b41e560917b270f2d151cf9a5f7c2439fde27aa0914"} Nov 01 20:48:17.528819 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:48:17.528761 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-c268d1/ipp-route-c268d1-kserve-d56f88c6f-wjs96" podStartSLOduration=1.528746927 podStartE2EDuration="1.528746927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:48:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:48:17.526407821 +0000 UTC m=+2943.299568333" watchObservedRunningTime="2026-11-01 20:48:17.528746927 +0000 UTC m=+2943.301907473" Nov 01 20:48:17.951923 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:48:17.951889 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-c268d1/ipp-route-c268d1-kserve-d56f88c6f-wjs96" Nov 01 20:48:26.951984 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:48:26.951951 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-c268d1/ipp-route-c268d1-kserve-d56f88c6f-wjs96" Nov 01 20:48:26.964581 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:48:26.964550 5935 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-c268d1/ipp-route-c268d1-kserve-d56f88c6f-wjs96" Nov 01 20:48:27.570651 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:48:27.570626 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-c268d1/ipp-route-c268d1-kserve-d56f88c6f-wjs96" Nov 01 20:48:35.844211 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:48:35.844174 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:48:57.156046 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:48:57.155968 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-c268d1/ipp-route-c268d1-kserve-d56f88c6f-wjs96"] Nov 01 20:49:05.843281 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:49:05.843243 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:49:12.178083 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:49:12.178024 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-c268d1/ipp-route-c268d1-kserve-d56f88c6f-wjs96" podUID="cf19ba79-a4e7-42bf-841e-47a105121445" containerName="main" containerID="cri-o://aa0b8d6b45da54e33112a528d928d25f32f8b1232a6d9711b360d464b7a7cfcd" gracePeriod=45 Nov 01 20:49:12.425936 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:49:12.425912 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-c268d1/ipp-route-c268d1-kserve-d56f88c6f-wjs96" Nov 01 20:49:12.513101 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:49:12.513068 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf19ba79-a4e7-42bf-841e-47a105121445-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf19ba79-a4e7-42bf-841e-47a105121445-dshm\") pod \"cf19ba79-a4e7-42bf-841e-47a105121445\" (UID: \"cf19ba79-a4e7-42bf-841e-47a105121445\") " Nov 01 20:49:12.513340 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:49:12.513126 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf19ba79-a4e7-42bf-841e-47a105121445-home\" (UniqueName: \"kubernetes.io/empty-dir/cf19ba79-a4e7-42bf-841e-47a105121445-home\") pod \"cf19ba79-a4e7-42bf-841e-47a105121445\" (UID: \"cf19ba79-a4e7-42bf-841e-47a105121445\") " Nov 01 20:49:12.513340 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:49:12.513200 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf19ba79-a4e7-42bf-841e-47a105121445-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf19ba79-a4e7-42bf-841e-47a105121445-model-cache\") pod \"cf19ba79-a4e7-42bf-841e-47a105121445\" (UID: \"cf19ba79-a4e7-42bf-841e-47a105121445\") " Nov 01 20:49:12.513340 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:49:12.513265 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf19ba79-a4e7-42bf-841e-47a105121445-tls-certs\" (UniqueName: \"kubernetes.io/secret/cf19ba79-a4e7-42bf-841e-47a105121445-tls-certs\") pod \"cf19ba79-a4e7-42bf-841e-47a105121445\" (UID: \"cf19ba79-a4e7-42bf-841e-47a105121445\") " Nov 01 20:49:12.513525 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:49:12.513340 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf19ba79-a4e7-42bf-841e-47a105121445-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf19ba79-a4e7-42bf-841e-47a105121445-tmp-dir\") pod \"cf19ba79-a4e7-42bf-841e-47a105121445\" (UID: \"cf19ba79-a4e7-42bf-841e-47a105121445\") " Nov 01 20:49:12.513525 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:49:12.513398 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf19ba79-a4e7-42bf-841e-47a105121445-kube-api-access-wmx8w\" (UniqueName: \"kubernetes.io/projected/cf19ba79-a4e7-42bf-841e-47a105121445-kube-api-access-wmx8w\") pod \"cf19ba79-a4e7-42bf-841e-47a105121445\" (UID: \"cf19ba79-a4e7-42bf-841e-47a105121445\") " Nov 01 20:49:12.513525 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:49:12.513384 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf19ba79-a4e7-42bf-841e-47a105121445-home" pod "cf19ba79-a4e7-42bf-841e-47a105121445" (UID: "cf19ba79-a4e7-42bf-841e-47a105121445"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:49:12.513525 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:49:12.513455 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf19ba79-a4e7-42bf-841e-47a105121445-model-cache" pod "cf19ba79-a4e7-42bf-841e-47a105121445" (UID: "cf19ba79-a4e7-42bf-841e-47a105121445"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:49:12.513733 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:49:12.513612 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf19ba79-a4e7-42bf-841e-47a105121445-tmp-dir" pod "cf19ba79-a4e7-42bf-841e-47a105121445" (UID: "cf19ba79-a4e7-42bf-841e-47a105121445"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:49:12.513733 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:49:12.513688 5935 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf19ba79-a4e7-42bf-841e-47a105121445-home\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:49:12.513733 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:49:12.513705 5935 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf19ba79-a4e7-42bf-841e-47a105121445-model-cache\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:49:12.513733 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:49:12.513718 5935 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf19ba79-a4e7-42bf-841e-47a105121445-tmp-dir\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:49:12.515696 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:49:12.515660 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf19ba79-a4e7-42bf-841e-47a105121445-kube-api-access-wmx8w" pod "cf19ba79-a4e7-42bf-841e-47a105121445" (UID: "cf19ba79-a4e7-42bf-841e-47a105121445"). InnerVolumeSpecName "kube-api-access-wmx8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:49:12.515824 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:49:12.515805 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf19ba79-a4e7-42bf-841e-47a105121445-dshm" pod "cf19ba79-a4e7-42bf-841e-47a105121445" (UID: "cf19ba79-a4e7-42bf-841e-47a105121445"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 01 20:49:12.515963 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:49:12.515936 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf19ba79-a4e7-42bf-841e-47a105121445-tls-certs" pod "cf19ba79-a4e7-42bf-841e-47a105121445" (UID: "cf19ba79-a4e7-42bf-841e-47a105121445"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:49:12.614673 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:49:12.614644 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmx8w\" (UniqueName: \"kubernetes.io/projected/cf19ba79-a4e7-42bf-841e-47a105121445-kube-api-access-wmx8w\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:49:12.614673 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:49:12.614668 5935 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf19ba79-a4e7-42bf-841e-47a105121445-dshm\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:49:12.614858 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:49:12.614682 5935 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf19ba79-a4e7-42bf-841e-47a105121445-tls-certs\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:49:12.782228 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:49:12.782148 5935 generic.go:356] "Generic (PLEG): container finished" podID="cf19ba79-a4e7-42bf-841e-47a105121445" containerID="aa0b8d6b45da54e33112a528d928d25f32f8b1232a6d9711b360d464b7a7cfcd" exitCode=0 Nov 01 20:49:12.782228 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:49:12.782186 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-c268d1/ipp-route-c268d1-kserve-d56f88c6f-wjs96" event={"ID":"cf19ba79-a4e7-42bf-841e-47a105121445","Type":"ContainerDied","Data":"aa0b8d6b45da54e33112a528d928d25f32f8b1232a6d9711b360d464b7a7cfcd"} Nov 01 20:49:12.782228 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:49:12.782192 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-c268d1/ipp-route-c268d1-kserve-d56f88c6f-wjs96" Nov 01 20:49:12.782508 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:49:12.782233 5935 scope.go:122] "RemoveContainer" containerID="aa0b8d6b45da54e33112a528d928d25f32f8b1232a6d9711b360d464b7a7cfcd" Nov 01 20:49:12.782508 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:49:12.782222 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-c268d1/ipp-route-c268d1-kserve-d56f88c6f-wjs96" event={"ID":"cf19ba79-a4e7-42bf-841e-47a105121445","Type":"ContainerDied","Data":"47c86c70c314b9e32afb3b41e560917b270f2d151cf9a5f7c2439fde27aa0914"} Nov 01 20:49:12.791206 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:49:12.791191 5935 scope.go:122] "RemoveContainer" containerID="aa0b8d6b45da54e33112a528d928d25f32f8b1232a6d9711b360d464b7a7cfcd" Nov 01 20:49:12.791523 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:49:12.791509 5935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa0b8d6b45da54e33112a528d928d25f32f8b1232a6d9711b360d464b7a7cfcd\": container with ID starting with aa0b8d6b45da54e33112a528d928d25f32f8b1232a6d9711b360d464b7a7cfcd not found: ID does not exist" containerID="aa0b8d6b45da54e33112a528d928d25f32f8b1232a6d9711b360d464b7a7cfcd" Nov 01 20:49:12.791579 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:49:12.791530 5935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa0b8d6b45da54e33112a528d928d25f32f8b1232a6d9711b360d464b7a7cfcd"} err="failed to get container status \"aa0b8d6b45da54e33112a528d928d25f32f8b1232a6d9711b360d464b7a7cfcd\": rpc error: code = NotFound desc = could not find container \"aa0b8d6b45da54e33112a528d928d25f32f8b1232a6d9711b360d464b7a7cfcd\": container with ID starting with aa0b8d6b45da54e33112a528d928d25f32f8b1232a6d9711b360d464b7a7cfcd not found: ID does not exist" Nov 01 20:49:12.801812 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:49:12.801787 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-c268d1/ipp-route-c268d1-kserve-d56f88c6f-wjs96"] Nov 01 20:49:12.807367 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:49:12.807338 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-c268d1/ipp-route-c268d1-kserve-d56f88c6f-wjs96"] Nov 01 20:49:14.352573 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:49:14.352539 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf19ba79-a4e7-42bf-841e-47a105121445" path="/var/lib/kubelet/pods/cf19ba79-a4e7-42bf-841e-47a105121445/volumes" Nov 01 20:49:35.843692 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:49:35.843653 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:50:05.844159 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:50:05.844116 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:50:25.018506 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:50:25.018431 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-724731db-lv9g8"] Nov 01 20:50:25.021257 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:50:25.021232 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-724731db-lv9g8"] Nov 01 20:50:26.351792 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:50:26.351757 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2969417c-c03a-486f-8b51-47d0f2ebf403" path="/var/lib/kubelet/pods/2969417c-c03a-486f-8b51-47d0f2ebf403/volumes" Nov 01 20:50:35.843064 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:50:35.843023 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:51:05.842661 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:51:05.842618 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:51:35.842980 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:51:35.842943 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:52:05.842397 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:52:05.842359 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:52:15.392842 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:52:15.392809 5935 scope.go:122] "RemoveContainer" containerID="1377f42472c1605728e3ecfd8f14ed7de3170947447bab51f8a1d31bcdd9e634" Nov 01 20:52:35.842862 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:52:35.842828 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:53:02.603476 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:53:02.603425 5935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-m6qv8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 20:53:02.604233 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:53:02.604184 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-m6qv8" podUID="9de91381-a29d-422d-b859-69229e8970c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:53:02.821519 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:53:02.821396 5935 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-fd4b99879-fl2rn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 01 20:53:02.821519 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:53:02.821508 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-fd4b99879-fl2rn" podUID="043fcc04-22eb-454f-a1d5-d669214c5b0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 01 20:53:05.842897 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:53:05.842860 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:53:35.862362 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:53:35.859225 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:54:05.843205 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:54:05.843169 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:54:35.842867 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:54:35.842827 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:55:05.843369 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:55:05.843335 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:55:35.844207 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:55:35.844169 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:56:00.976238 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:56:00.976206 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4e9e72-d776d5967dhxq2"] Nov 01 20:56:00.976751 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:56:00.976585 5935 cpu_manager.go:414] "containerMap: removing container" podUID="cf19ba79-a4e7-42bf-841e-47a105121445" containerName="main" Nov 01 20:56:00.976751 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:56:00.976596 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf19ba79-a4e7-42bf-841e-47a105121445" containerName="main" Nov 01 20:56:00.976751 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:56:00.976668 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf19ba79-a4e7-42bf-841e-47a105121445" containerName="main" Nov 01 20:56:00.979898 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:56:00.979873 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e9e72-d776d5967dhxq2" Nov 01 20:56:00.988431 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:56:00.988394 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4e9e72-d776d5967dhxq2"] Nov 01 20:56:01.013892 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:56:01.013864 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-bbf535b9-dockercfg-vr8x9\"" Nov 01 20:56:01.063404 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:56:01.063379 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-4e9e72\"" Nov 01 20:56:01.094464 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:56:01.094433 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/975ca969-37d8-414e-be02-b1723ea6b1e7-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-4e9e72-d776d5967dhxq2\" (UID: \"975ca969-37d8-414e-be02-b1723ea6b1e7\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e9e72-d776d5967dhxq2" Nov 01 20:56:01.094596 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:56:01.094473 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7ck2\" (UniqueName: \"kubernetes.io/projected/975ca969-37d8-414e-be02-b1723ea6b1e7-kube-api-access-t7ck2\") pod \"payload-processing-e2e-worker-single-main-4e9e72-d776d5967dhxq2\" (UID: \"975ca969-37d8-414e-be02-b1723ea6b1e7\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e9e72-d776d5967dhxq2" Nov 01 20:56:01.195820 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:56:01.195781 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7ck2\" (UniqueName: \"kubernetes.io/projected/975ca969-37d8-414e-be02-b1723ea6b1e7-kube-api-access-t7ck2\") pod \"payload-processing-e2e-worker-single-main-4e9e72-d776d5967dhxq2\" (UID: \"975ca969-37d8-414e-be02-b1723ea6b1e7\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e9e72-d776d5967dhxq2" Nov 01 20:56:01.196023 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:56:01.195917 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/975ca969-37d8-414e-be02-b1723ea6b1e7-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-4e9e72-d776d5967dhxq2\" (UID: \"975ca969-37d8-414e-be02-b1723ea6b1e7\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e9e72-d776d5967dhxq2" Nov 01 20:56:01.196624 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:56:01.196605 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/975ca969-37d8-414e-be02-b1723ea6b1e7-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-4e9e72-d776d5967dhxq2\" (UID: \"975ca969-37d8-414e-be02-b1723ea6b1e7\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e9e72-d776d5967dhxq2" Nov 01 20:56:01.204034 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:56:01.204003 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7ck2\" (UniqueName: \"kubernetes.io/projected/975ca969-37d8-414e-be02-b1723ea6b1e7-kube-api-access-t7ck2\") pod \"payload-processing-e2e-worker-single-main-4e9e72-d776d5967dhxq2\" (UID: \"975ca969-37d8-414e-be02-b1723ea6b1e7\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e9e72-d776d5967dhxq2" Nov 01 20:56:01.298259 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:56:01.298182 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e9e72-d776d5967dhxq2" Nov 01 20:56:01.575187 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:56:01.575166 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4e9e72-d776d5967dhxq2"] Nov 01 20:56:01.577685 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:56:01.577653 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod975ca969_37d8_414e_be02_b1723ea6b1e7.slice/crio-c210993abdbdbece5030feeb6647a16207e89fdd06bae3a4200552dae55fd1bd WatchSource:0}: Error finding container c210993abdbdbece5030feeb6647a16207e89fdd06bae3a4200552dae55fd1bd: Status 404 returned error can't find the container with id c210993abdbdbece5030feeb6647a16207e89fdd06bae3a4200552dae55fd1bd Nov 01 20:56:01.981054 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:56:01.981018 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e9e72-d776d5967dhxq2" event={"ID":"975ca969-37d8-414e-be02-b1723ea6b1e7","Type":"ContainerStarted","Data":"262b4050fee737e04f2bbd71b4dc188d58c8649887d32b841ee9e13f79606a71"} Nov 01 20:56:01.981472 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:56:01.981082 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e9e72-d776d5967dhxq2" Nov 01 20:56:01.981472 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:56:01.981104 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e9e72-d776d5967dhxq2" event={"ID":"975ca969-37d8-414e-be02-b1723ea6b1e7","Type":"ContainerStarted","Data":"c210993abdbdbece5030feeb6647a16207e89fdd06bae3a4200552dae55fd1bd"} Nov 01 20:56:01.997990 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:56:01.997944 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e9e72-d776d5967dhxq2" podStartSLOduration=1.997930024 podStartE2EDuration="1.997930024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:56:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:56:01.996801209 +0000 UTC m=+3407.769961720" watchObservedRunningTime="2026-11-01 20:56:01.997930024 +0000 UTC m=+3407.771090577" Nov 01 20:56:05.846263 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:56:05.846224 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:56:12.989385 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:56:12.989357 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e9e72-d776d5967dhxq2" Nov 01 20:56:35.844455 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:56:35.844371 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:57:02.938268 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:57:02.938236 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4e9e72-d776d5967dhxq2"] Nov 01 20:57:05.843736 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:57:05.843691 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:57:07.938943 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:57:07.938894 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e9e72-d776d5967dhxq2" podUID="975ca969-37d8-414e-be02-b1723ea6b1e7" containerName="payload-processing" containerID="cri-o://262b4050fee737e04f2bbd71b4dc188d58c8649887d32b841ee9e13f79606a71" gracePeriod=25 Nov 01 20:57:12.987574 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:57:12.987541 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e9e72-d776d5967dhxq2" podUID="975ca969-37d8-414e-be02-b1723ea6b1e7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 20:57:22.987885 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:57:22.987848 5935 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e9e72-d776d5967dhxq2" podUID="975ca969-37d8-414e-be02-b1723ea6b1e7" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 01 20:57:28.181800 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:57:28.181775 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e9e72-d776d5967dhxq2" Nov 01 20:57:28.263856 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:57:28.263772 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/975ca969-37d8-414e-be02-b1723ea6b1e7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/975ca969-37d8-414e-be02-b1723ea6b1e7-plugins-config-volume\") pod \"975ca969-37d8-414e-be02-b1723ea6b1e7\" (UID: \"975ca969-37d8-414e-be02-b1723ea6b1e7\") " Nov 01 20:57:28.264025 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:57:28.263873 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/975ca969-37d8-414e-be02-b1723ea6b1e7-kube-api-access-t7ck2\" (UniqueName: \"kubernetes.io/projected/975ca969-37d8-414e-be02-b1723ea6b1e7-kube-api-access-t7ck2\") pod \"975ca969-37d8-414e-be02-b1723ea6b1e7\" (UID: \"975ca969-37d8-414e-be02-b1723ea6b1e7\") " Nov 01 20:57:28.264147 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:57:28.264124 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/975ca969-37d8-414e-be02-b1723ea6b1e7-plugins-config-volume" pod "975ca969-37d8-414e-be02-b1723ea6b1e7" (UID: "975ca969-37d8-414e-be02-b1723ea6b1e7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:57:28.265898 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:57:28.265876 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/975ca969-37d8-414e-be02-b1723ea6b1e7-kube-api-access-t7ck2" pod "975ca969-37d8-414e-be02-b1723ea6b1e7" (UID: "975ca969-37d8-414e-be02-b1723ea6b1e7"). InnerVolumeSpecName "kube-api-access-t7ck2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:57:28.365526 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:57:28.365503 5935 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/975ca969-37d8-414e-be02-b1723ea6b1e7-plugins-config-volume\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:57:28.365526 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:57:28.365526 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7ck2\" (UniqueName: \"kubernetes.io/projected/975ca969-37d8-414e-be02-b1723ea6b1e7-kube-api-access-t7ck2\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:57:28.407171 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:57:28.407140 5935 generic.go:356] "Generic (PLEG): container finished" podID="975ca969-37d8-414e-be02-b1723ea6b1e7" containerID="262b4050fee737e04f2bbd71b4dc188d58c8649887d32b841ee9e13f79606a71" exitCode=0 Nov 01 20:57:28.407316 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:57:28.407195 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e9e72-d776d5967dhxq2" Nov 01 20:57:28.407316 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:57:28.407231 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e9e72-d776d5967dhxq2" event={"ID":"975ca969-37d8-414e-be02-b1723ea6b1e7","Type":"ContainerDied","Data":"262b4050fee737e04f2bbd71b4dc188d58c8649887d32b841ee9e13f79606a71"} Nov 01 20:57:28.407457 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:57:28.407318 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e9e72-d776d5967dhxq2" event={"ID":"975ca969-37d8-414e-be02-b1723ea6b1e7","Type":"ContainerDied","Data":"c210993abdbdbece5030feeb6647a16207e89fdd06bae3a4200552dae55fd1bd"} Nov 01 20:57:28.407457 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:57:28.407351 5935 scope.go:122] "RemoveContainer" containerID="262b4050fee737e04f2bbd71b4dc188d58c8649887d32b841ee9e13f79606a71" Nov 01 20:57:28.415861 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:57:28.415843 5935 scope.go:122] "RemoveContainer" containerID="262b4050fee737e04f2bbd71b4dc188d58c8649887d32b841ee9e13f79606a71" Nov 01 20:57:28.416113 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:57:28.416094 5935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"262b4050fee737e04f2bbd71b4dc188d58c8649887d32b841ee9e13f79606a71\": container with ID starting with 262b4050fee737e04f2bbd71b4dc188d58c8649887d32b841ee9e13f79606a71 not found: ID does not exist" containerID="262b4050fee737e04f2bbd71b4dc188d58c8649887d32b841ee9e13f79606a71" Nov 01 20:57:28.416182 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:57:28.416123 5935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"262b4050fee737e04f2bbd71b4dc188d58c8649887d32b841ee9e13f79606a71"} err="failed to get container status \"262b4050fee737e04f2bbd71b4dc188d58c8649887d32b841ee9e13f79606a71\": rpc error: code = NotFound desc = could not find container \"262b4050fee737e04f2bbd71b4dc188d58c8649887d32b841ee9e13f79606a71\": container with ID starting with 262b4050fee737e04f2bbd71b4dc188d58c8649887d32b841ee9e13f79606a71 not found: ID does not exist" Nov 01 20:57:28.422784 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:57:28.422766 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4e9e72-d776d5967dhxq2"] Nov 01 20:57:28.426647 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:57:28.426626 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4e9e72-d776d5967dhxq2"] Nov 01 20:57:30.351394 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:57:30.351364 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="975ca969-37d8-414e-be02-b1723ea6b1e7" path="/var/lib/kubelet/pods/975ca969-37d8-414e-be02-b1723ea6b1e7/volumes" Nov 01 20:57:35.418210 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:57:35.418177 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4e9e72-d776d5967rfj57"] Nov 01 20:57:35.418610 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:57:35.418598 5935 cpu_manager.go:414] "containerMap: removing container" podUID="975ca969-37d8-414e-be02-b1723ea6b1e7" containerName="payload-processing" Nov 01 20:57:35.418658 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:57:35.418610 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="975ca969-37d8-414e-be02-b1723ea6b1e7" containerName="payload-processing" Nov 01 20:57:35.418693 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:57:35.418680 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="975ca969-37d8-414e-be02-b1723ea6b1e7" containerName="payload-processing" Nov 01 20:57:35.423193 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:57:35.423172 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e9e72-d776d5967rfj57" Nov 01 20:57:35.432434 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:57:35.432396 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-4e9e72\"" Nov 01 20:57:35.433501 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:57:35.433476 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4e9e72-d776d5967rfj57"] Nov 01 20:57:35.530321 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:57:35.530261 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9438b09-f02b-4a7c-b072-1e3d5de2b0d8-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-4e9e72-d776d5967rfj57\" (UID: \"f9438b09-f02b-4a7c-b072-1e3d5de2b0d8\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e9e72-d776d5967rfj57" Nov 01 20:57:35.530489 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:57:35.530389 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67qj7\" (UniqueName: \"kubernetes.io/projected/f9438b09-f02b-4a7c-b072-1e3d5de2b0d8-kube-api-access-67qj7\") pod \"payload-processing-e2e-worker-single-main-4e9e72-d776d5967rfj57\" (UID: \"f9438b09-f02b-4a7c-b072-1e3d5de2b0d8\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e9e72-d776d5967rfj57" Nov 01 20:57:35.631562 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:57:35.631525 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9438b09-f02b-4a7c-b072-1e3d5de2b0d8-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-4e9e72-d776d5967rfj57\" (UID: \"f9438b09-f02b-4a7c-b072-1e3d5de2b0d8\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e9e72-d776d5967rfj57" Nov 01 20:57:35.631734 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:57:35.631573 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67qj7\" (UniqueName: \"kubernetes.io/projected/f9438b09-f02b-4a7c-b072-1e3d5de2b0d8-kube-api-access-67qj7\") pod \"payload-processing-e2e-worker-single-main-4e9e72-d776d5967rfj57\" (UID: \"f9438b09-f02b-4a7c-b072-1e3d5de2b0d8\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e9e72-d776d5967rfj57" Nov 01 20:57:35.632225 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:57:35.632206 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9438b09-f02b-4a7c-b072-1e3d5de2b0d8-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-4e9e72-d776d5967rfj57\" (UID: \"f9438b09-f02b-4a7c-b072-1e3d5de2b0d8\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e9e72-d776d5967rfj57" Nov 01 20:57:35.639304 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:57:35.639265 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67qj7\" (UniqueName: \"kubernetes.io/projected/f9438b09-f02b-4a7c-b072-1e3d5de2b0d8-kube-api-access-67qj7\") pod \"payload-processing-e2e-worker-single-main-4e9e72-d776d5967rfj57\" (UID: \"f9438b09-f02b-4a7c-b072-1e3d5de2b0d8\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e9e72-d776d5967rfj57" Nov 01 20:57:35.751272 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:57:35.751236 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e9e72-d776d5967rfj57" Nov 01 20:57:35.851261 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:57:35.851218 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:57:36.025771 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:57:36.025745 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4e9e72-d776d5967rfj57"] Nov 01 20:57:36.028259 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:57:36.028230 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9438b09_f02b_4a7c_b072_1e3d5de2b0d8.slice/crio-f3f30f54996b67b7efc6557b4d77ab0a16e3f985caa8e1b5087b367f3154d7f8 WatchSource:0}: Error finding container f3f30f54996b67b7efc6557b4d77ab0a16e3f985caa8e1b5087b367f3154d7f8: Status 404 returned error can't find the container with id f3f30f54996b67b7efc6557b4d77ab0a16e3f985caa8e1b5087b367f3154d7f8 Nov 01 20:57:36.451621 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:57:36.451585 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e9e72-d776d5967rfj57" event={"ID":"f9438b09-f02b-4a7c-b072-1e3d5de2b0d8","Type":"ContainerStarted","Data":"3a287b3e0df551d45b62b396482dd094ae7a9c61ab2f9cda746926ff51c6011d"} Nov 01 20:57:36.452002 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:57:36.451647 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e9e72-d776d5967rfj57" event={"ID":"f9438b09-f02b-4a7c-b072-1e3d5de2b0d8","Type":"ContainerStarted","Data":"f3f30f54996b67b7efc6557b4d77ab0a16e3f985caa8e1b5087b367f3154d7f8"} Nov 01 20:57:36.452002 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:57:36.451686 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e9e72-d776d5967rfj57" Nov 01 20:57:36.469478 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:57:36.469433 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e9e72-d776d5967rfj57" podStartSLOduration=1.469420836 podStartE2EDuration="1.469420836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:57:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:57:36.468340764 +0000 UTC m=+3502.241501277" watchObservedRunningTime="2026-11-01 20:57:36.469420836 +0000 UTC m=+3502.242581344" Nov 01 20:57:47.460967 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:57:47.460936 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e9e72-d776d5967rfj57" Nov 01 20:57:54.297892 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:57:54.297843 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4e9e72-d776d5967rfj57"] Nov 01 20:57:59.298252 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:57:59.298165 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e9e72-d776d5967rfj57" podUID="f9438b09-f02b-4a7c-b072-1e3d5de2b0d8" containerName="payload-processing" containerID="cri-o://3a287b3e0df551d45b62b396482dd094ae7a9c61ab2f9cda746926ff51c6011d" gracePeriod=25 Nov 01 20:57:59.539512 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:57:59.539487 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e9e72-d776d5967rfj57" Nov 01 20:57:59.576779 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:57:59.576714 5935 generic.go:356] "Generic (PLEG): container finished" podID="f9438b09-f02b-4a7c-b072-1e3d5de2b0d8" containerID="3a287b3e0df551d45b62b396482dd094ae7a9c61ab2f9cda746926ff51c6011d" exitCode=0 Nov 01 20:57:59.576893 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:57:59.576789 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e9e72-d776d5967rfj57" event={"ID":"f9438b09-f02b-4a7c-b072-1e3d5de2b0d8","Type":"ContainerDied","Data":"3a287b3e0df551d45b62b396482dd094ae7a9c61ab2f9cda746926ff51c6011d"} Nov 01 20:57:59.576893 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:57:59.576813 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e9e72-d776d5967rfj57" Nov 01 20:57:59.576893 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:57:59.576839 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-4e9e72-d776d5967rfj57" event={"ID":"f9438b09-f02b-4a7c-b072-1e3d5de2b0d8","Type":"ContainerDied","Data":"f3f30f54996b67b7efc6557b4d77ab0a16e3f985caa8e1b5087b367f3154d7f8"} Nov 01 20:57:59.576893 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:57:59.576867 5935 scope.go:122] "RemoveContainer" containerID="3a287b3e0df551d45b62b396482dd094ae7a9c61ab2f9cda746926ff51c6011d" Nov 01 20:57:59.589853 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:57:59.589821 5935 scope.go:122] "RemoveContainer" containerID="3a287b3e0df551d45b62b396482dd094ae7a9c61ab2f9cda746926ff51c6011d" Nov 01 20:57:59.590220 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:57:59.590200 5935 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a287b3e0df551d45b62b396482dd094ae7a9c61ab2f9cda746926ff51c6011d\": container with ID starting with 3a287b3e0df551d45b62b396482dd094ae7a9c61ab2f9cda746926ff51c6011d not found: ID does not exist" containerID="3a287b3e0df551d45b62b396482dd094ae7a9c61ab2f9cda746926ff51c6011d" Nov 01 20:57:59.590333 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:57:59.590233 5935 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a287b3e0df551d45b62b396482dd094ae7a9c61ab2f9cda746926ff51c6011d"} err="failed to get container status \"3a287b3e0df551d45b62b396482dd094ae7a9c61ab2f9cda746926ff51c6011d\": rpc error: code = NotFound desc = could not find container \"3a287b3e0df551d45b62b396482dd094ae7a9c61ab2f9cda746926ff51c6011d\": container with ID starting with 3a287b3e0df551d45b62b396482dd094ae7a9c61ab2f9cda746926ff51c6011d not found: ID does not exist" Nov 01 20:57:59.652673 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:57:59.652648 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9438b09-f02b-4a7c-b072-1e3d5de2b0d8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9438b09-f02b-4a7c-b072-1e3d5de2b0d8-plugins-config-volume\") pod \"f9438b09-f02b-4a7c-b072-1e3d5de2b0d8\" (UID: \"f9438b09-f02b-4a7c-b072-1e3d5de2b0d8\") " Nov 01 20:57:59.652775 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:57:59.652687 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9438b09-f02b-4a7c-b072-1e3d5de2b0d8-kube-api-access-67qj7\" (UniqueName: \"kubernetes.io/projected/f9438b09-f02b-4a7c-b072-1e3d5de2b0d8-kube-api-access-67qj7\") pod \"f9438b09-f02b-4a7c-b072-1e3d5de2b0d8\" (UID: \"f9438b09-f02b-4a7c-b072-1e3d5de2b0d8\") " Nov 01 20:57:59.653003 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:57:59.652979 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9438b09-f02b-4a7c-b072-1e3d5de2b0d8-plugins-config-volume" pod "f9438b09-f02b-4a7c-b072-1e3d5de2b0d8" (UID: "f9438b09-f02b-4a7c-b072-1e3d5de2b0d8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:57:59.654669 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:57:59.654652 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9438b09-f02b-4a7c-b072-1e3d5de2b0d8-kube-api-access-67qj7" pod "f9438b09-f02b-4a7c-b072-1e3d5de2b0d8" (UID: "f9438b09-f02b-4a7c-b072-1e3d5de2b0d8"). InnerVolumeSpecName "kube-api-access-67qj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:57:59.753731 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:57:59.753709 5935 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f9438b09-f02b-4a7c-b072-1e3d5de2b0d8-plugins-config-volume\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:57:59.753731 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:57:59.753732 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67qj7\" (UniqueName: \"kubernetes.io/projected/f9438b09-f02b-4a7c-b072-1e3d5de2b0d8-kube-api-access-67qj7\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:57:59.897623 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:57:59.897601 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4e9e72-d776d5967rfj57"] Nov 01 20:57:59.901452 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:57:59.901426 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-4e9e72-d776d5967rfj57"] Nov 01 20:58:00.352041 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:00.352011 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9438b09-f02b-4a7c-b072-1e3d5de2b0d8" path="/var/lib/kubelet/pods/f9438b09-f02b-4a7c-b072-1e3d5de2b0d8/volumes" Nov 01 20:58:05.842271 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:58:05.842234 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:58:28.608919 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:28.608884 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-8f9dc8cf-64flj"] Nov 01 20:58:28.609567 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:28.609467 5935 cpu_manager.go:414] "containerMap: removing container" podUID="f9438b09-f02b-4a7c-b072-1e3d5de2b0d8" containerName="payload-processing" Nov 01 20:58:28.609567 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:28.609488 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9438b09-f02b-4a7c-b072-1e3d5de2b0d8" containerName="payload-processing" Nov 01 20:58:28.609691 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:28.609597 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9438b09-f02b-4a7c-b072-1e3d5de2b0d8" containerName="payload-processing" Nov 01 20:58:28.612746 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:28.612727 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8f9dc8cf-64flj" Nov 01 20:58:28.622061 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:28.622037 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 01 20:58:28.622268 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:28.622240 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8f9dc8cf-64flj"] Nov 01 20:58:28.622491 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:28.622467 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7vx47\"" Nov 01 20:58:28.622491 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:28.622474 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 01 20:58:28.644538 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:28.644513 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c937a5a6-96bf-43ba-b753-1730ecc04929-openshift-service-ca\") pod \"maas-api-8f9dc8cf-64flj\" (UID: \"c937a5a6-96bf-43ba-b753-1730ecc04929\") " pod="odh-ai-gateway-infra/maas-api-8f9dc8cf-64flj" Nov 01 20:58:28.644672 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:28.644557 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c937a5a6-96bf-43ba-b753-1730ecc04929-maas-api-tls\") pod \"maas-api-8f9dc8cf-64flj\" (UID: \"c937a5a6-96bf-43ba-b753-1730ecc04929\") " pod="odh-ai-gateway-infra/maas-api-8f9dc8cf-64flj" Nov 01 20:58:28.644672 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:28.644631 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2845\" (UniqueName: \"kubernetes.io/projected/c937a5a6-96bf-43ba-b753-1730ecc04929-kube-api-access-j2845\") pod \"maas-api-8f9dc8cf-64flj\" (UID: \"c937a5a6-96bf-43ba-b753-1730ecc04929\") " pod="odh-ai-gateway-infra/maas-api-8f9dc8cf-64flj" Nov 01 20:58:28.644796 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:28.644677 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c937a5a6-96bf-43ba-b753-1730ecc04929-metrics-certs\") pod \"maas-api-8f9dc8cf-64flj\" (UID: \"c937a5a6-96bf-43ba-b753-1730ecc04929\") " pod="odh-ai-gateway-infra/maas-api-8f9dc8cf-64flj" Nov 01 20:58:28.745246 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:28.745219 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c937a5a6-96bf-43ba-b753-1730ecc04929-metrics-certs\") pod \"maas-api-8f9dc8cf-64flj\" (UID: \"c937a5a6-96bf-43ba-b753-1730ecc04929\") " pod="odh-ai-gateway-infra/maas-api-8f9dc8cf-64flj" Nov 01 20:58:28.745435 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:28.745277 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c937a5a6-96bf-43ba-b753-1730ecc04929-openshift-service-ca\") pod \"maas-api-8f9dc8cf-64flj\" (UID: \"c937a5a6-96bf-43ba-b753-1730ecc04929\") " pod="odh-ai-gateway-infra/maas-api-8f9dc8cf-64flj" Nov 01 20:58:28.745435 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:28.745324 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c937a5a6-96bf-43ba-b753-1730ecc04929-maas-api-tls\") pod \"maas-api-8f9dc8cf-64flj\" (UID: \"c937a5a6-96bf-43ba-b753-1730ecc04929\") " pod="odh-ai-gateway-infra/maas-api-8f9dc8cf-64flj" Nov 01 20:58:28.745435 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:28.745418 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2845\" (UniqueName: \"kubernetes.io/projected/c937a5a6-96bf-43ba-b753-1730ecc04929-kube-api-access-j2845\") pod \"maas-api-8f9dc8cf-64flj\" (UID: \"c937a5a6-96bf-43ba-b753-1730ecc04929\") " pod="odh-ai-gateway-infra/maas-api-8f9dc8cf-64flj" Nov 01 20:58:28.746028 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:28.746009 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c937a5a6-96bf-43ba-b753-1730ecc04929-openshift-service-ca\") pod \"maas-api-8f9dc8cf-64flj\" (UID: \"c937a5a6-96bf-43ba-b753-1730ecc04929\") " pod="odh-ai-gateway-infra/maas-api-8f9dc8cf-64flj" Nov 01 20:58:28.747581 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:28.747559 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c937a5a6-96bf-43ba-b753-1730ecc04929-maas-api-tls\") pod \"maas-api-8f9dc8cf-64flj\" (UID: \"c937a5a6-96bf-43ba-b753-1730ecc04929\") " pod="odh-ai-gateway-infra/maas-api-8f9dc8cf-64flj" Nov 01 20:58:28.747696 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:28.747677 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c937a5a6-96bf-43ba-b753-1730ecc04929-metrics-certs\") pod \"maas-api-8f9dc8cf-64flj\" (UID: \"c937a5a6-96bf-43ba-b753-1730ecc04929\") " pod="odh-ai-gateway-infra/maas-api-8f9dc8cf-64flj" Nov 01 20:58:28.752495 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:28.752474 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2845\" (UniqueName: \"kubernetes.io/projected/c937a5a6-96bf-43ba-b753-1730ecc04929-kube-api-access-j2845\") pod \"maas-api-8f9dc8cf-64flj\" (UID: \"c937a5a6-96bf-43ba-b753-1730ecc04929\") " pod="odh-ai-gateway-infra/maas-api-8f9dc8cf-64flj" Nov 01 20:58:28.927556 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:28.927480 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8f9dc8cf-64flj" Nov 01 20:58:29.199029 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:29.198977 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8f9dc8cf-64flj"] Nov 01 20:58:29.201358 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:58:29.201329 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc937a5a6_96bf_43ba_b753_1730ecc04929.slice/crio-b191fd78f14ca7497bc353a42b48ec182e97c96808bb2b29c6dc5377159b95d1 WatchSource:0}: Error finding container b191fd78f14ca7497bc353a42b48ec182e97c96808bb2b29c6dc5377159b95d1: Status 404 returned error can't find the container with id b191fd78f14ca7497bc353a42b48ec182e97c96808bb2b29c6dc5377159b95d1 Nov 01 20:58:29.202582 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:29.202565 5935 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 01 20:58:29.751353 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:29.751326 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8f9dc8cf-64flj" event={"ID":"c937a5a6-96bf-43ba-b753-1730ecc04929","Type":"ContainerStarted","Data":"b191fd78f14ca7497bc353a42b48ec182e97c96808bb2b29c6dc5377159b95d1"} Nov 01 20:58:30.758192 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:30.758156 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8f9dc8cf-64flj" event={"ID":"c937a5a6-96bf-43ba-b753-1730ecc04929","Type":"ContainerStarted","Data":"ab04ebd6d80bb593d8cec6efbcf8bbc58c169009630c8ad2d3f5b7e588914b56"} Nov 01 20:58:30.758659 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:30.758213 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-8f9dc8cf-64flj" Nov 01 20:58:30.779155 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:30.779096 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-8f9dc8cf-64flj" podStartSLOduration=2.2707797149999998 podStartE2EDuration="2.779080057s" totalImagesPullingTime="508.300342ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:58:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:58:30.775057436 +0000 UTC m=+3556.548217949" watchObservedRunningTime="2026-11-01 20:58:30.779080057 +0000 UTC m=+3556.552240571" Nov 01 20:58:35.842957 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:58:35.842918 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:58:36.549358 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:36.549317 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-8f9dc8cf-64flj"] Nov 01 20:58:36.549617 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:36.549566 5935 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-8f9dc8cf-64flj" podUID="c937a5a6-96bf-43ba-b753-1730ecc04929" containerName="maas-api" containerID="cri-o://ab04ebd6d80bb593d8cec6efbcf8bbc58c169009630c8ad2d3f5b7e588914b56" gracePeriod=30 Nov 01 20:58:36.555918 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:36.555895 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-8f9dc8cf-64flj" Nov 01 20:58:36.583891 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:36.583867 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6b5986f9c5-7ptvh"] Nov 01 20:58:36.587280 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:36.587264 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b5986f9c5-7ptvh" Nov 01 20:58:36.599685 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:36.599660 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b5986f9c5-7ptvh"] Nov 01 20:58:36.715132 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:36.715099 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8703b36-0b08-49c2-bcb0-2e20a09d01a8-maas-api-tls\") pod \"maas-api-6b5986f9c5-7ptvh\" (UID: \"d8703b36-0b08-49c2-bcb0-2e20a09d01a8\") " pod="odh-ai-gateway-infra/maas-api-6b5986f9c5-7ptvh" Nov 01 20:58:36.715323 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:36.715158 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8703b36-0b08-49c2-bcb0-2e20a09d01a8-openshift-service-ca\") pod \"maas-api-6b5986f9c5-7ptvh\" (UID: \"d8703b36-0b08-49c2-bcb0-2e20a09d01a8\") " pod="odh-ai-gateway-infra/maas-api-6b5986f9c5-7ptvh" Nov 01 20:58:36.715410 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:36.715344 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8703b36-0b08-49c2-bcb0-2e20a09d01a8-metrics-certs\") pod \"maas-api-6b5986f9c5-7ptvh\" (UID: \"d8703b36-0b08-49c2-bcb0-2e20a09d01a8\") " pod="odh-ai-gateway-infra/maas-api-6b5986f9c5-7ptvh" Nov 01 20:58:36.715470 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:36.715451 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cq4c\" (UniqueName: \"kubernetes.io/projected/d8703b36-0b08-49c2-bcb0-2e20a09d01a8-kube-api-access-6cq4c\") pod \"maas-api-6b5986f9c5-7ptvh\" (UID: \"d8703b36-0b08-49c2-bcb0-2e20a09d01a8\") " pod="odh-ai-gateway-infra/maas-api-6b5986f9c5-7ptvh" Nov 01 20:58:36.791090 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:36.791057 5935 generic.go:356] "Generic (PLEG): container finished" podID="c937a5a6-96bf-43ba-b753-1730ecc04929" containerID="ab04ebd6d80bb593d8cec6efbcf8bbc58c169009630c8ad2d3f5b7e588914b56" exitCode=0 Nov 01 20:58:36.791226 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:36.791105 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8f9dc8cf-64flj" event={"ID":"c937a5a6-96bf-43ba-b753-1730ecc04929","Type":"ContainerDied","Data":"ab04ebd6d80bb593d8cec6efbcf8bbc58c169009630c8ad2d3f5b7e588914b56"} Nov 01 20:58:36.791226 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:36.791170 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8f9dc8cf-64flj" event={"ID":"c937a5a6-96bf-43ba-b753-1730ecc04929","Type":"ContainerDied","Data":"b191fd78f14ca7497bc353a42b48ec182e97c96808bb2b29c6dc5377159b95d1"} Nov 01 20:58:36.791226 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:36.791194 5935 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b191fd78f14ca7497bc353a42b48ec182e97c96808bb2b29c6dc5377159b95d1" Nov 01 20:58:36.800825 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:36.800774 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8f9dc8cf-64flj" Nov 01 20:58:36.815994 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:36.815969 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8703b36-0b08-49c2-bcb0-2e20a09d01a8-metrics-certs\") pod \"maas-api-6b5986f9c5-7ptvh\" (UID: \"d8703b36-0b08-49c2-bcb0-2e20a09d01a8\") " pod="odh-ai-gateway-infra/maas-api-6b5986f9c5-7ptvh" Nov 01 20:58:36.816097 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:36.816034 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cq4c\" (UniqueName: \"kubernetes.io/projected/d8703b36-0b08-49c2-bcb0-2e20a09d01a8-kube-api-access-6cq4c\") pod \"maas-api-6b5986f9c5-7ptvh\" (UID: \"d8703b36-0b08-49c2-bcb0-2e20a09d01a8\") " pod="odh-ai-gateway-infra/maas-api-6b5986f9c5-7ptvh" Nov 01 20:58:36.816159 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:36.816093 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8703b36-0b08-49c2-bcb0-2e20a09d01a8-maas-api-tls\") pod \"maas-api-6b5986f9c5-7ptvh\" (UID: \"d8703b36-0b08-49c2-bcb0-2e20a09d01a8\") " pod="odh-ai-gateway-infra/maas-api-6b5986f9c5-7ptvh" Nov 01 20:58:36.816159 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:36.816134 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8703b36-0b08-49c2-bcb0-2e20a09d01a8-openshift-service-ca\") pod \"maas-api-6b5986f9c5-7ptvh\" (UID: \"d8703b36-0b08-49c2-bcb0-2e20a09d01a8\") " pod="odh-ai-gateway-infra/maas-api-6b5986f9c5-7ptvh" Nov 01 20:58:36.816868 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:36.816844 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8703b36-0b08-49c2-bcb0-2e20a09d01a8-openshift-service-ca\") pod \"maas-api-6b5986f9c5-7ptvh\" (UID: \"d8703b36-0b08-49c2-bcb0-2e20a09d01a8\") " pod="odh-ai-gateway-infra/maas-api-6b5986f9c5-7ptvh" Nov 01 20:58:36.818718 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:36.818698 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8703b36-0b08-49c2-bcb0-2e20a09d01a8-metrics-certs\") pod \"maas-api-6b5986f9c5-7ptvh\" (UID: \"d8703b36-0b08-49c2-bcb0-2e20a09d01a8\") " pod="odh-ai-gateway-infra/maas-api-6b5986f9c5-7ptvh" Nov 01 20:58:36.819116 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:36.819091 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8703b36-0b08-49c2-bcb0-2e20a09d01a8-maas-api-tls\") pod \"maas-api-6b5986f9c5-7ptvh\" (UID: \"d8703b36-0b08-49c2-bcb0-2e20a09d01a8\") " pod="odh-ai-gateway-infra/maas-api-6b5986f9c5-7ptvh" Nov 01 20:58:36.825122 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:36.825100 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cq4c\" (UniqueName: \"kubernetes.io/projected/d8703b36-0b08-49c2-bcb0-2e20a09d01a8-kube-api-access-6cq4c\") pod \"maas-api-6b5986f9c5-7ptvh\" (UID: \"d8703b36-0b08-49c2-bcb0-2e20a09d01a8\") " pod="odh-ai-gateway-infra/maas-api-6b5986f9c5-7ptvh" Nov 01 20:58:36.917373 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:36.917346 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c937a5a6-96bf-43ba-b753-1730ecc04929-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c937a5a6-96bf-43ba-b753-1730ecc04929-openshift-service-ca\") pod \"c937a5a6-96bf-43ba-b753-1730ecc04929\" (UID: \"c937a5a6-96bf-43ba-b753-1730ecc04929\") " Nov 01 20:58:36.917749 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:36.917476 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c937a5a6-96bf-43ba-b753-1730ecc04929-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c937a5a6-96bf-43ba-b753-1730ecc04929-metrics-certs\") pod \"c937a5a6-96bf-43ba-b753-1730ecc04929\" (UID: \"c937a5a6-96bf-43ba-b753-1730ecc04929\") " Nov 01 20:58:36.917749 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:36.917536 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c937a5a6-96bf-43ba-b753-1730ecc04929-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c937a5a6-96bf-43ba-b753-1730ecc04929-maas-api-tls\") pod \"c937a5a6-96bf-43ba-b753-1730ecc04929\" (UID: \"c937a5a6-96bf-43ba-b753-1730ecc04929\") " Nov 01 20:58:36.917749 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:36.917599 5935 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c937a5a6-96bf-43ba-b753-1730ecc04929-kube-api-access-j2845\" (UniqueName: \"kubernetes.io/projected/c937a5a6-96bf-43ba-b753-1730ecc04929-kube-api-access-j2845\") pod \"c937a5a6-96bf-43ba-b753-1730ecc04929\" (UID: \"c937a5a6-96bf-43ba-b753-1730ecc04929\") " Nov 01 20:58:36.917749 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:36.917701 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c937a5a6-96bf-43ba-b753-1730ecc04929-openshift-service-ca" pod "c937a5a6-96bf-43ba-b753-1730ecc04929" (UID: "c937a5a6-96bf-43ba-b753-1730ecc04929"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 01 20:58:36.918050 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:36.918028 5935 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c937a5a6-96bf-43ba-b753-1730ecc04929-openshift-service-ca\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:58:36.919617 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:36.919592 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c937a5a6-96bf-43ba-b753-1730ecc04929-maas-api-tls" pod "c937a5a6-96bf-43ba-b753-1730ecc04929" (UID: "c937a5a6-96bf-43ba-b753-1730ecc04929"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:58:36.919706 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:36.919676 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c937a5a6-96bf-43ba-b753-1730ecc04929-metrics-certs" pod "c937a5a6-96bf-43ba-b753-1730ecc04929" (UID: "c937a5a6-96bf-43ba-b753-1730ecc04929"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 01 20:58:36.919814 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:36.919795 5935 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c937a5a6-96bf-43ba-b753-1730ecc04929-kube-api-access-j2845" pod "c937a5a6-96bf-43ba-b753-1730ecc04929" (UID: "c937a5a6-96bf-43ba-b753-1730ecc04929"). InnerVolumeSpecName "kube-api-access-j2845". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 01 20:58:36.929942 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:36.929922 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6b5986f9c5-7ptvh" Nov 01 20:58:37.019091 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:37.018976 5935 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c937a5a6-96bf-43ba-b753-1730ecc04929-metrics-certs\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:58:37.019091 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:37.019015 5935 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c937a5a6-96bf-43ba-b753-1730ecc04929-maas-api-tls\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:58:37.019091 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:37.019032 5935 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2845\" (UniqueName: \"kubernetes.io/projected/c937a5a6-96bf-43ba-b753-1730ecc04929-kube-api-access-j2845\") on node \"ip-10-0-136-26.us-east-2.compute.internal\" DevicePath \"\"" Nov 01 20:58:37.224742 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:37.224714 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6b5986f9c5-7ptvh"] Nov 01 20:58:37.227144 ip-10-0-136-26 kubenswrapper[5935]: W1101 20:58:37.227097 5935 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8703b36_0b08_49c2_bcb0_2e20a09d01a8.slice/crio-a8330491f389dac991aeb082fee7b934dafb153833f89d6fac000411519b65a9 WatchSource:0}: Error finding container a8330491f389dac991aeb082fee7b934dafb153833f89d6fac000411519b65a9: Status 404 returned error can't find the container with id a8330491f389dac991aeb082fee7b934dafb153833f89d6fac000411519b65a9 Nov 01 20:58:37.799236 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:37.799200 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b5986f9c5-7ptvh" event={"ID":"d8703b36-0b08-49c2-bcb0-2e20a09d01a8","Type":"ContainerStarted","Data":"32305dc2c2710c6e272907801a140425d8055135b462703471eed637a52ef614"} Nov 01 20:58:37.799447 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:37.799255 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6b5986f9c5-7ptvh" event={"ID":"d8703b36-0b08-49c2-bcb0-2e20a09d01a8","Type":"ContainerStarted","Data":"a8330491f389dac991aeb082fee7b934dafb153833f89d6fac000411519b65a9"} Nov 01 20:58:37.799447 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:37.799352 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6b5986f9c5-7ptvh" Nov 01 20:58:37.799578 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:37.799478 5935 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8f9dc8cf-64flj" Nov 01 20:58:37.816480 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:37.816424 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6b5986f9c5-7ptvh" podStartSLOduration=1.388039056 podStartE2EDuration="1.816404835s" totalImagesPullingTime="428.365779ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 20:58:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 20:58:37.813888446 +0000 UTC m=+3563.587048960" watchObservedRunningTime="2026-11-01 20:58:37.816404835 +0000 UTC m=+3563.589565347" Nov 01 20:58:37.827654 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:37.827623 5935 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-8f9dc8cf-64flj"] Nov 01 20:58:37.829667 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:37.829612 5935 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-8f9dc8cf-64flj"] Nov 01 20:58:38.361310 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:38.357951 5935 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c937a5a6-96bf-43ba-b753-1730ecc04929" path="/var/lib/kubelet/pods/c937a5a6-96bf-43ba-b753-1730ecc04929/volumes" Nov 01 20:58:43.816407 ip-10-0-136-26 kubenswrapper[5935]: I1101 20:58:43.816373 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6b5986f9c5-7ptvh" Nov 01 20:59:05.842993 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:59:05.842957 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 20:59:35.843103 ip-10-0-136-26 kubenswrapper[5935]: E1101 20:59:35.843023 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 21:00:05.851987 ip-10-0-136-26 kubenswrapper[5935]: E1101 21:00:05.851938 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 21:00:35.844848 ip-10-0-136-26 kubenswrapper[5935]: E1101 21:00:35.844809 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 21:01:01.312070 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:01.312037 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-x9p74_e4de1911-27fd-43cb-970b-308f8380bd6d/manager/0.log" Nov 01 21:01:01.676274 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:01.676182 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-vtpsz_458588d6-f7a6-4e19-a2e5-75bf2a9373fc/manager/0.log" Nov 01 21:01:02.149325 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:02.149298 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-jbfsn_33b28b24-2fcc-40d9-9ad0-059f15acf4aa/server/0.log" Nov 01 21:01:02.537641 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:02.537605 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-9hj4l_557bbd53-195b-4a00-985e-d3f8647758d1/proxy/0.log" Nov 01 21:01:03.408173 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:03.408139 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6b5986f9c5-7ptvh_d8703b36-0b08-49c2-bcb0-2e20a09d01a8/maas-api/0.log" Nov 01 21:01:05.437092 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:05.437061 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-hrbm5_69c13cd0-756b-4cc2-9eae-20a1ac83f22d/postgres/0.log" Nov 01 21:01:05.844126 ip-10-0-136-26 kubenswrapper[5935]: E1101 21:01:05.844083 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 21:01:09.487067 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:09.487027 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-79f98cb86fbgwkn_87d067a6-b0e8-46d7-92f8-df20c468a672/istio-proxy/0.log" Nov 01 21:01:10.719904 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:10.719868 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-fl2rn_043fcc04-22eb-454f-a1d5-d669214c5b0c/istio-proxy/0.log" Nov 01 21:01:11.089250 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:11.089157 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-ljtcq_03014616-4520-48bb-a548-66485577e26c/istio-proxy/1.log" Nov 01 21:01:11.276972 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:11.276940 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-m6qv8_9de91381-a29d-422d-b859-69229e8970c7/istio-proxy/0.log" Nov 01 21:01:13.082163 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:13.082122 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-ljtcq_03014616-4520-48bb-a548-66485577e26c/istio-proxy/0.log" Nov 01 21:01:27.797770 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:27.797740 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-k5brd_2efc5bbf-efd4-4a42-bed8-53f2490bb7bc/global-pull-secret-syncer/0.log" Nov 01 21:01:28.044887 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:28.044851 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-rz7h2_8537e78c-5b2c-4966-9558-044ddc39119b/konnectivity-agent/0.log" Nov 01 21:01:28.095352 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:28.095248 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-26.us-east-2.compute.internal_d535588bb4025d791b22a311a0d6e9f5/haproxy/0.log" Nov 01 21:01:33.776261 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:33.776222 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-x9p74_e4de1911-27fd-43cb-970b-308f8380bd6d/manager/0.log" Nov 01 21:01:34.800685 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:34.800642 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-vtpsz_458588d6-f7a6-4e19-a2e5-75bf2a9373fc/manager/0.log" Nov 01 21:01:34.976539 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:34.976503 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-jbfsn_33b28b24-2fcc-40d9-9ad0-059f15acf4aa/server/0.log" Nov 01 21:01:35.844144 ip-10-0-136-26 kubenswrapper[5935]: E1101 21:01:35.844104 5935 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-r4spt" containerName="extractor" Nov 01 21:01:37.108103 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:37.108072 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-z8k8x_961c6886-e297-4202-b10a-df97f3537736/node-exporter/0.log" Nov 01 21:01:37.128843 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:37.128813 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-z8k8x_961c6886-e297-4202-b10a-df97f3537736/kube-rbac-proxy/0.log" Nov 01 21:01:37.150886 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:37.150861 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-z8k8x_961c6886-e297-4202-b10a-df97f3537736/init-textfile/0.log" Nov 01 21:01:37.271488 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:37.271447 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-9hj4l_557bbd53-195b-4a00-985e-d3f8647758d1/proxy/0.log" Nov 01 21:01:37.271747 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:37.271727 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9/prometheus/0.log" Nov 01 21:01:37.293220 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:37.293195 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9/config-reloader/0.log" Nov 01 21:01:37.320264 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:37.320238 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9/thanos-sidecar/0.log" Nov 01 21:01:37.341799 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:37.341772 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9/kube-rbac-proxy-web/0.log" Nov 01 21:01:37.363644 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:37.363556 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9/kube-rbac-proxy/0.log" Nov 01 21:01:37.385818 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:37.385785 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9/kube-rbac-proxy-thanos/0.log" Nov 01 21:01:37.413132 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:37.413102 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4d7a2e00-cce9-4bc2-baf8-a27bfe9107b9/init-config-reloader/0.log" Nov 01 21:01:37.436482 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:37.436446 5935 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-z9gbm/perf-node-gather-daemonset-xh58l"] Nov 01 21:01:37.436947 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:37.436926 5935 cpu_manager.go:414] "containerMap: removing container" podUID="c937a5a6-96bf-43ba-b753-1730ecc04929" containerName="maas-api" Nov 01 21:01:37.436947 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:37.436948 5935 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c937a5a6-96bf-43ba-b753-1730ecc04929" containerName="maas-api" Nov 01 21:01:37.437205 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:37.437036 5935 memory_manager.go:365] "RemoveStaleState removing state" podUID="c937a5a6-96bf-43ba-b753-1730ecc04929" containerName="maas-api" Nov 01 21:01:37.440470 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:37.440448 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-z9gbm/perf-node-gather-daemonset-xh58l" Nov 01 21:01:37.450158 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:37.450132 5935 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-z9gbm\"/\"default-dockercfg-fndqt\"" Nov 01 21:01:37.450336 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:37.450149 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-z9gbm\"/\"kube-root-ca.crt\"" Nov 01 21:01:37.450449 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:37.450226 5935 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-z9gbm\"/\"openshift-service-ca.crt\"" Nov 01 21:01:37.451812 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:37.451787 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-z9gbm/perf-node-gather-daemonset-xh58l"] Nov 01 21:01:37.462224 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:37.462173 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-6tbtd_fa82c276-5679-40f4-b7b4-e73eba7de9ef/prometheus-operator/0.log" Nov 01 21:01:37.484844 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:37.484794 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-6tbtd_fa82c276-5679-40f4-b7b4-e73eba7de9ef/kube-rbac-proxy/0.log" Nov 01 21:01:37.500368 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:37.500338 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6405239c-7b7d-4512-aab3-bef82899782a-lib-modules\") pod \"perf-node-gather-daemonset-xh58l\" (UID: \"6405239c-7b7d-4512-aab3-bef82899782a\") " pod="openshift-must-gather-z9gbm/perf-node-gather-daemonset-xh58l" Nov 01 21:01:37.500545 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:37.500375 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6405239c-7b7d-4512-aab3-bef82899782a-proc\") pod \"perf-node-gather-daemonset-xh58l\" (UID: \"6405239c-7b7d-4512-aab3-bef82899782a\") " pod="openshift-must-gather-z9gbm/perf-node-gather-daemonset-xh58l" Nov 01 21:01:37.500545 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:37.500395 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xksb6\" (UniqueName: \"kubernetes.io/projected/6405239c-7b7d-4512-aab3-bef82899782a-kube-api-access-xksb6\") pod \"perf-node-gather-daemonset-xh58l\" (UID: \"6405239c-7b7d-4512-aab3-bef82899782a\") " pod="openshift-must-gather-z9gbm/perf-node-gather-daemonset-xh58l" Nov 01 21:01:37.500545 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:37.500525 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6405239c-7b7d-4512-aab3-bef82899782a-podres\") pod \"perf-node-gather-daemonset-xh58l\" (UID: \"6405239c-7b7d-4512-aab3-bef82899782a\") " pod="openshift-must-gather-z9gbm/perf-node-gather-daemonset-xh58l" Nov 01 21:01:37.500659 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:37.500562 5935 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6405239c-7b7d-4512-aab3-bef82899782a-sys\") pod \"perf-node-gather-daemonset-xh58l\" (UID: \"6405239c-7b7d-4512-aab3-bef82899782a\") " pod="openshift-must-gather-z9gbm/perf-node-gather-daemonset-xh58l" Nov 01 21:01:37.601936 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:37.601898 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6405239c-7b7d-4512-aab3-bef82899782a-podres\") pod \"perf-node-gather-daemonset-xh58l\" (UID: \"6405239c-7b7d-4512-aab3-bef82899782a\") " pod="openshift-must-gather-z9gbm/perf-node-gather-daemonset-xh58l" Nov 01 21:01:37.602135 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:37.601965 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6405239c-7b7d-4512-aab3-bef82899782a-sys\") pod \"perf-node-gather-daemonset-xh58l\" (UID: \"6405239c-7b7d-4512-aab3-bef82899782a\") " pod="openshift-must-gather-z9gbm/perf-node-gather-daemonset-xh58l" Nov 01 21:01:37.602135 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:37.602029 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6405239c-7b7d-4512-aab3-bef82899782a-lib-modules\") pod \"perf-node-gather-daemonset-xh58l\" (UID: \"6405239c-7b7d-4512-aab3-bef82899782a\") " pod="openshift-must-gather-z9gbm/perf-node-gather-daemonset-xh58l" Nov 01 21:01:37.602135 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:37.602046 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6405239c-7b7d-4512-aab3-bef82899782a-podres\") pod \"perf-node-gather-daemonset-xh58l\" (UID: \"6405239c-7b7d-4512-aab3-bef82899782a\") " pod="openshift-must-gather-z9gbm/perf-node-gather-daemonset-xh58l" Nov 01 21:01:37.602135 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:37.602062 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6405239c-7b7d-4512-aab3-bef82899782a-proc\") pod \"perf-node-gather-daemonset-xh58l\" (UID: \"6405239c-7b7d-4512-aab3-bef82899782a\") " pod="openshift-must-gather-z9gbm/perf-node-gather-daemonset-xh58l" Nov 01 21:01:37.602135 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:37.602092 5935 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xksb6\" (UniqueName: \"kubernetes.io/projected/6405239c-7b7d-4512-aab3-bef82899782a-kube-api-access-xksb6\") pod \"perf-node-gather-daemonset-xh58l\" (UID: \"6405239c-7b7d-4512-aab3-bef82899782a\") " pod="openshift-must-gather-z9gbm/perf-node-gather-daemonset-xh58l" Nov 01 21:01:37.602135 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:37.602094 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6405239c-7b7d-4512-aab3-bef82899782a-sys\") pod \"perf-node-gather-daemonset-xh58l\" (UID: \"6405239c-7b7d-4512-aab3-bef82899782a\") " pod="openshift-must-gather-z9gbm/perf-node-gather-daemonset-xh58l" Nov 01 21:01:37.602135 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:37.602127 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6405239c-7b7d-4512-aab3-bef82899782a-lib-modules\") pod \"perf-node-gather-daemonset-xh58l\" (UID: \"6405239c-7b7d-4512-aab3-bef82899782a\") " pod="openshift-must-gather-z9gbm/perf-node-gather-daemonset-xh58l" Nov 01 21:01:37.602384 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:37.602160 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6405239c-7b7d-4512-aab3-bef82899782a-proc\") pod \"perf-node-gather-daemonset-xh58l\" (UID: \"6405239c-7b7d-4512-aab3-bef82899782a\") " pod="openshift-must-gather-z9gbm/perf-node-gather-daemonset-xh58l" Nov 01 21:01:37.609545 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:37.609513 5935 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xksb6\" (UniqueName: \"kubernetes.io/projected/6405239c-7b7d-4512-aab3-bef82899782a-kube-api-access-xksb6\") pod \"perf-node-gather-daemonset-xh58l\" (UID: \"6405239c-7b7d-4512-aab3-bef82899782a\") " pod="openshift-must-gather-z9gbm/perf-node-gather-daemonset-xh58l" Nov 01 21:01:37.617248 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:37.617155 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-596fbdd49c-2nmrq_40d57e7e-4896-4e0c-b266-ef5e6ca8beab/thanos-query/0.log" Nov 01 21:01:37.642139 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:37.642108 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-596fbdd49c-2nmrq_40d57e7e-4896-4e0c-b266-ef5e6ca8beab/kube-rbac-proxy-web/0.log" Nov 01 21:01:37.674644 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:37.674614 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-596fbdd49c-2nmrq_40d57e7e-4896-4e0c-b266-ef5e6ca8beab/kube-rbac-proxy/0.log" Nov 01 21:01:37.702856 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:37.702826 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-596fbdd49c-2nmrq_40d57e7e-4896-4e0c-b266-ef5e6ca8beab/prom-label-proxy/0.log" Nov 01 21:01:37.724583 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:37.724551 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-596fbdd49c-2nmrq_40d57e7e-4896-4e0c-b266-ef5e6ca8beab/kube-rbac-proxy-rules/0.log" Nov 01 21:01:37.746757 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:37.746723 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-596fbdd49c-2nmrq_40d57e7e-4896-4e0c-b266-ef5e6ca8beab/kube-rbac-proxy-metrics/0.log" Nov 01 21:01:37.754808 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:37.754775 5935 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-z9gbm/perf-node-gather-daemonset-xh58l" Nov 01 21:01:38.031704 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:38.031668 5935 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-z9gbm/perf-node-gather-daemonset-xh58l"] Nov 01 21:01:38.169542 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:38.169513 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-x9p74_e4de1911-27fd-43cb-970b-308f8380bd6d/manager/0.log" Nov 01 21:01:38.853562 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:38.853480 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z9gbm/perf-node-gather-daemonset-xh58l" event={"ID":"6405239c-7b7d-4512-aab3-bef82899782a","Type":"ContainerStarted","Data":"1ce33ac2516e308a136f55a0cea23fb727f39d48ca1fc53fc8840bff433d0cdc"} Nov 01 21:01:38.853562 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:38.853546 5935 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-z9gbm/perf-node-gather-daemonset-xh58l" event={"ID":"6405239c-7b7d-4512-aab3-bef82899782a","Type":"ContainerStarted","Data":"3ba7d6acb8be1a1f799d20912e9c2c09b00fa0e83a4f8d2576f988c12037c5c7"} Nov 01 21:01:38.853981 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:38.853803 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-z9gbm/perf-node-gather-daemonset-xh58l" Nov 01 21:01:38.871013 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:38.870953 5935 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-z9gbm/perf-node-gather-daemonset-xh58l" podStartSLOduration=1.870934706 podStartE2EDuration="1.870934706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 21:01:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 21:01:38.868357082 +0000 UTC m=+3744.641517595" watchObservedRunningTime="2026-11-01 21:01:38.870934706 +0000 UTC m=+3744.644095221" Nov 01 21:01:39.157317 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:39.157222 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-vtpsz_458588d6-f7a6-4e19-a2e5-75bf2a9373fc/manager/0.log" Nov 01 21:01:39.358571 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:39.358544 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-jbfsn_33b28b24-2fcc-40d9-9ad0-059f15acf4aa/server/0.log" Nov 01 21:01:40.091680 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:40.091648 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5fccf54c48-xhbhp_d871b82e-7d5c-4faf-8918-74cf2b465c82/console/0.log" Nov 01 21:01:41.629227 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:41.629195 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-9hj4l_557bbd53-195b-4a00-985e-d3f8647758d1/proxy/0.log" Nov 01 21:01:41.896007 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:41.895930 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5lztx_58167352-076f-40d7-abee-1c5f48f67566/dns/0.log" Nov 01 21:01:41.919103 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:41.919069 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5lztx_58167352-076f-40d7-abee-1c5f48f67566/kube-rbac-proxy/0.log" Nov 01 21:01:42.103308 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:42.103251 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-tzgt2_c0e3a247-a792-431d-85cb-9b46982c3373/dns-node-resolver/0.log" Nov 01 21:01:42.438154 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:42.438125 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-x9p74_e4de1911-27fd-43cb-970b-308f8380bd6d/manager/0.log" Nov 01 21:01:42.724585 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:42.724547 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-77fcl_983a921d-61a5-4668-8e17-1a24b6eaf8b6/node-ca/0.log" Nov 01 21:01:43.437990 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:43.437955 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-vtpsz_458588d6-f7a6-4e19-a2e5-75bf2a9373fc/manager/0.log" Nov 01 21:01:43.633856 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:43.633819 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-jbfsn_33b28b24-2fcc-40d9-9ad0-059f15acf4aa/server/0.log" Nov 01 21:01:43.766768 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:43.766725 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-79f98cb86fbgwkn_87d067a6-b0e8-46d7-92f8-df20c468a672/istio-proxy/0.log" Nov 01 21:01:44.636594 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:44.636557 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-fl2rn_043fcc04-22eb-454f-a1d5-d669214c5b0c/istio-proxy/0.log" Nov 01 21:01:44.689482 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:44.689438 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-ljtcq_03014616-4520-48bb-a548-66485577e26c/istio-proxy/0.log" Nov 01 21:01:44.696970 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:44.696933 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-ljtcq_03014616-4520-48bb-a548-66485577e26c/istio-proxy/1.log" Nov 01 21:01:44.727807 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:44.727777 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-m6qv8_9de91381-a29d-422d-b859-69229e8970c7/istio-proxy/0.log" Nov 01 21:01:44.868518 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:44.868494 5935 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-z9gbm/perf-node-gather-daemonset-xh58l" Nov 01 21:01:46.485809 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:46.485777 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-9hj4l_557bbd53-195b-4a00-985e-d3f8647758d1/proxy/0.log" Nov 01 21:01:46.755224 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:46.755144 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-tq7l5_a225113f-cd49-48a9-b211-595bf5de763a/serve-healthcheck-canary/0.log" Nov 01 21:01:47.509322 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:47.509276 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-79f98cb86fbgwkn_87d067a6-b0e8-46d7-92f8-df20c468a672/istio-proxy/0.log" Nov 01 21:01:47.635859 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:47.635824 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-r4spt_83a0642b-4dfe-4ddb-b506-f0654c7b6d30/kube-rbac-proxy/0.log" Nov 01 21:01:47.658089 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:47.658056 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-r4spt_83a0642b-4dfe-4ddb-b506-f0654c7b6d30/exporter/0.log" Nov 01 21:01:47.694332 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:47.694300 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-r4spt_83a0642b-4dfe-4ddb-b506-f0654c7b6d30/extractor/0.log" Nov 01 21:01:47.807875 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:47.807770 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-fl2rn_043fcc04-22eb-454f-a1d5-d669214c5b0c/istio-proxy/0.log" Nov 01 21:01:47.844249 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:47.844213 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-ljtcq_03014616-4520-48bb-a548-66485577e26c/istio-proxy/0.log" Nov 01 21:01:47.851825 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:47.851793 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-ljtcq_03014616-4520-48bb-a548-66485577e26c/istio-proxy/1.log" Nov 01 21:01:47.873199 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:47.873168 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-m6qv8_9de91381-a29d-422d-b859-69229e8970c7/istio-proxy/0.log" Nov 01 21:01:48.030066 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:48.030032 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-79f98cb86fbgwkn_87d067a6-b0e8-46d7-92f8-df20c468a672/istio-proxy/0.log" Nov 01 21:01:48.361867 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:48.361830 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-fl2rn_043fcc04-22eb-454f-a1d5-d669214c5b0c/istio-proxy/0.log" Nov 01 21:01:48.410511 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:48.410473 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-ljtcq_03014616-4520-48bb-a548-66485577e26c/istio-proxy/0.log" Nov 01 21:01:48.418022 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:48.417991 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-ljtcq_03014616-4520-48bb-a548-66485577e26c/istio-proxy/1.log" Nov 01 21:01:48.444358 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:48.444325 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-m6qv8_9de91381-a29d-422d-b859-69229e8970c7/istio-proxy/0.log" Nov 01 21:01:50.716876 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:50.716841 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-689bcf9f9-lvm4r_48a12663-97c4-486c-9462-183947bbfdc1/manager/0.log" Nov 01 21:01:50.738305 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:50.738262 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-689bcf9f9-lvm4r_48a12663-97c4-486c-9462-183947bbfdc1/copy-manifests/0.log" Nov 01 21:01:51.365216 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:51.365187 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-79f98cb86fbgwkn_87d067a6-b0e8-46d7-92f8-df20c468a672/istio-proxy/0.log" Nov 01 21:01:51.642933 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:51.642854 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-fl2rn_043fcc04-22eb-454f-a1d5-d669214c5b0c/istio-proxy/0.log" Nov 01 21:01:51.677552 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:51.677524 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-ljtcq_03014616-4520-48bb-a548-66485577e26c/istio-proxy/0.log" Nov 01 21:01:51.684994 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:51.684945 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-ljtcq_03014616-4520-48bb-a548-66485577e26c/istio-proxy/1.log" Nov 01 21:01:51.704472 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:51.704445 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-fd4b99879-m6qv8_9de91381-a29d-422d-b859-69229e8970c7/istio-proxy/0.log" Nov 01 21:01:51.981193 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:51.981163 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-jobset-operator_jobset-controller-manager-7bbf46bdb7-57pw7_05b8db82-93af-4b55-9454-d1e0df8a9aee/manager/0.log" Nov 01 21:01:52.008250 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:52.008221 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-jobset-operator_jobset-operator-674cd59dc5-mcmdt_c68e7c85-46f6-408d-8296-bcdbd72ba817/jobset-operator/0.log" Nov 01 21:01:52.707682 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:52.707640 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-x9p74_e4de1911-27fd-43cb-970b-308f8380bd6d/manager/0.log" Nov 01 21:01:53.703581 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:53.703546 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-vtpsz_458588d6-f7a6-4e19-a2e5-75bf2a9373fc/manager/0.log" Nov 01 21:01:53.883825 ip-10-0-136-26 kubenswrapper[5935]: I1101 21:01:53.883794 5935 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-jbfsn_33b28b24-2fcc-40d9-9ad0-059f15acf4aa/server/0.log"